Blog
0%

Posted on

本文主要研究一下MySql 1215的错误: “Cannot add foreign key constraint”

我们在设计多表关联时，经常会需要用到外键，也就是当前表引用其他表的主键。但有时候哪里写的不对，MySQL会提示1215错误。但我们并不知道是什么原因导致的错误。

我们来分析常见的情况:

表或者约束引用的索引不存在(通常出现在载入存储文件)

如何诊断: 运行SHOW TABLE或者SHOW CREATE TABLE。如果出现1146错误，意味着表没有按顺序创建
如何解决: 运行CREATE TABLE或者临时禁用外键

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
mysql> CREATE TABLE child (
    ->   id INT(10) NOT NULL PRIMARY KEY,
    ->   parent_id INT(10),
    ->   FOREIGN KEY (parent_id) REFERENCES `parent`(`id`)
    -> ) ENGINE INNODB;
ERROR 1215 (HY000): Cannot add foreign key constraint
# We check for the parent table and is not there.
mysql> SHOW TABLES LIKE 'par%';
Empty set (0.00 sec)
# We go ahead and create the parent table (we’ll use the same parent table structure for all other example in this blogpost):
mysql> CREATE TABLE parent (
    ->   id INT(10) NOT NULL PRIMARY KEY,
    ->   column_1 INT(10) NOT NULL,
    ->   column_2 INT(10) NOT NULL,
    ->   column_3 INT(10) NOT NULL,
    ->   column_4 CHAR(10) CHARACTER SET utf8 COLLATE utf8_bin,
    ->   KEY column_2_column_3_idx (column_2, column_3),
    ->   KEY column_4_idx (column_4)
    -> ) ENGINE INNODB;
Query OK, 0 rows affected (0.00 sec)
# And now we re-attempt to create the child table
mysql> CREATE TABLE child (
    ->   id INT(10) NOT NULL PRIMARY KEY,drop table child;
    ->   parent_id INT(10),
    ->   FOREIGN KEY (parent_id) REFERENCES `parent`(`id`)
    -> ) ENGINE INNODB;
Query OK, 0 rows affected (0.01 sec)

表或者约束引用的索引错误引号

如何诊断: 检查所有的外键，确保所有的引用正确
如何解决: 加上缺少的引号

例子:

1
2
3
4
5
6
7
8
9
10
11
# wrong; single pair of backticks wraps both table and column
ALTER TABLE child  ADD FOREIGN KEY (parent_id) REFERENCES `parent(id)`;
 
# correct; one pair for each part
ALTER TABLE child ADD FOREIGN KEY (parent_id) REFERENCES `parent`(`id`);
 
# also correct; no backticks anywhere
ALTER TABLE child ADD FOREIGN KEY (parent_id) REFERENCES parent(id);
 
# also correct; backticks on either object (in case it’s a keyword)
ALTER TABLE child ADD FOREIGN KEY (parent_id) REFERENCES parent(`id`);

这里需要注意引用表需要加上对应的列名，不能因为当前表的列名与引用表的主键名一致而忽略。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
create table department
	(dept_name		varchar(20), 
	 building		varchar(15), 
	 budget		        numeric(12,2) check (budget > 0),
	 primary key (dept_name)
	);
create table course
	(course_id		varchar(8), 
	 title			varchar(50), 
	 dept_name		varchar(20),
	 credits		numeric(2,0) check (credits > 0),
	 primary key (course_id),
	 foreign key (dept_name) references department(dept_name)
		on delete set null
	) ;

这里如果references后面的表明没有写上列名，会报错

约束引用的键或表名称写错

如何诊断: 运行SHOW TABLES和SHOW COLUMNS比较声明的引用
如何解决: 写上正确的名称

例子:

1
2
3
4
5
# wrong; Parent table name is ‘parent’
ALTER TABLE child ADD FOREIGN KEY (parent_id) REFERENCES pariente(id);
 
# correct
ALTER TABLE child ADD FOREIGN KEY (parent_id) REFERENCES parent(id);

约束引用的外键类型和长度不一致

如何诊断: 运行SHOW CREATE TABLE parent检查所有引用类型和长度一致
如何解决: 修改引用类型和长度

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
# wrong; id column in parent is INT(10)
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY, 
  parent_id BIGINT(10) NOT NULL,
  FOREIGN KEY (parent_id) REFERENCES `parent`(`id`)
) ENGINE INNODB; 
 
# correct; id column matches definition of parent table
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY, 
  parent_id INT(10) NOT NULL, 
  FOREIGN KEY (parent_id) REFERENCES `parent`(`id`)
) ENGINE INNODB;

外键不是KEY类型的任意一种

如何诊断: 使用SHOW CREATE TABLE parent来检查引用的列是正确的
如何解决: 使引用的列加上KEY,UNIQUE KEY或者PRIMARY KEY的一种

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# wrong; column_1 is not indexed in our example table
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_1 INT(10),
  FOREIGN KEY (parent_column_1) REFERENCES `parent`(`column_1`)
) ENGINE INNODB;
# correct; we first add an index and then re-attempt creation of child table
ALTER TABLE parent ADD INDEX column_1_idx(column_1);
# and then re-attempt creation of child table
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_1 INT(10),
  FOREIGN KEY (parent_column_1) REFERENCES `parent`(`column_1`)
) ENGINE INNODB;

引用多个外键

如何诊断:运行SHOW CREATE TABLE parent检查多引用外键的正确性
如何解决:给最左的键加上索引

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# wrong; column_3 only appears as the second part of an index on parent table
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_3 INT(10),
  FOREIGN KEY (parent_column_3) REFERENCES `parent`(`column_3`)
) ENGINE INNODB;
 
# correct; create a new index for the referenced column
ALTER TABLE parent ADD INDEX column_3_idx (column_3);
 
# then re-attempt creation of child
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_3 INT(10),
  FOREIGN KEY (parent_column_3) REFERENCES `parent`(`column_3`)
) ENGINE INNODB;

两个表或者列使用不同的字符集或者排序

如何诊断: 比较父类表和子类表的字符集和排序
如何解决: 修改子类的字符集和排序

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
# wrong; the parent table uses utf8/utf8_bin for charset/collation 
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_4 CHAR(10) CHARACTER SET utf8 COLLATE utf8_unicode_ci,
  FOREIGN KEY (parent_column_4) REFERENCES `parent`(`column_4`)
) ENGINE INNODB;
 
# correct; edited DDL so COLLATE matches parent definition
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_column_4 CHAR(10) CHARACTER SET utf8 COLLATE utf8_bin,
  FOREIGN KEY (parent_column_4) REFERENCES `parent`(`column_4`)
) ENGINE INNODB;

父类表没有使用InnoDB

如何诊断: 运行SHOW CREATE TABLE parent，检查是否是InnoDB
如何修复:修改父类表，使其采用InnoDB

例子:

1
2
3
4
5
6
# wrong; the parent table in this example is MyISAM:
CREATE TABLE parent (
  id INT(10) NOT NULL PRIMARY KEY
) ENGINE MyISAM;
# correct: we modify the parent’s engine
ALTER TABLE parent ENGINE=INNODB;

使用简短语法引用外键

如何诊断: 检查引用部分, MySQL不支持简短语法
如何解决: 修改子类表，规定引用的表和列

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
# wrong; only parent table name is specified in REFERENCES
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  column_2 INT(10) NOT NULL,
  FOREIGN KEY (column_2) REFERENCES parent
) ENGINE INNODB;
 
# correct; both the table and column are in the REFERENCES definition
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  column_2 INT(10) NOT NULL,
  FOREIGN KEY (column_2) REFERENCES parent(column_2)
) ENGINE INNODB;

父表被分割了

如何诊断: 检查父表是否被分割
如何修复: 合并被分割的部分

例子:

1
2
3
4
5
6
7
8
9
# wrong: the parent table we see below is using PARTITIONs
CREATE TABLE parent (
  id INT(10) NOT NULL PRIMARY KEY
) ENGINE INNODB
PARTITION BY HASH(id)
PARTITIONS 6;
 
#correct: ALTER parent table to remove partitioning
ALTER TABLE parent REMOVE PARTITIONING;

引用的列是虚列

如何诊断: 检查引用的列是否是虚列
如何修复: 修改父表的列，确保列不是虚列

例子:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# wrong; this parent table has a generated virtual column
CREATE TABLE parent (
  id INT(10) NOT NULL PRIMARY KEY,
  column_1 INT(10) NOT NULL,
  column_2 INT(10) NOT NULL,
  column_virt INT(10) AS (column_1 + column_2) NOT NULL,
  KEY column_virt_idx (column_virt)
) ENGINE INNODB;
 
# correct: make the column STORED so it can be used as a foreign key
ALTER TABLE parent DROP COLUMN column_virt, ADD COLUMN column_virt INT(10) AS (column_1 + column_2) STORED NOT NULL;
 
# And now the child table can be created pointing to column_virt
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_virt INT(10) NOT NULL,
  FOREIGN KEY (parent_virt) REFERENCES parent(column_virt)
) ENGINE INNODB;

对约束行为使用默认集

如何诊断: 检查使用约束行为的字表的行为，尝试使用SET DEFAULT
如何修复: 移除SET DEFAULT

例子

1
2
3
4
5
6
7
8
9
10
11
12
13
# wrong; the constraint action uses SET DEFAULT
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_id INT(10) NOT NULL,
  FOREIGN KEY (parent_id) REFERENCES parent(id) ON UPDATE SET DEFAULT
) ENGINE INNODB;        
 
# correct; there's no alternative to SET DEFAULT, removing or picking other is the corrective measure
CREATE TABLE child (
  id INT(10) NOT NULL PRIMARY KEY,
  parent_id INT(10) NOT NULL,
  FOREIGN KEY (parent_id) REFERENCES parent(id) 
) ENGINE INNODB;

总结

通过以上的这些诊断基本上能够解决1215这个错误。关于外键的更多信息可以参考Mysql外键约束

参考: Dealing with MySQL Error Code 1215: “Cannot add foreign key constraint”

Posted on

Eureka是什么？

Eureka is a REST (Representational State Transfer) based service that is primarily used in the AWS cloud for locating services for the purpose of load balancing and failover of middle-tier servers. We call this service, the Eureka Server. Eureka also comes with a Java-based client component,the Eureka Client, which makes interactions with the service much easier. The client also has a built-in load balancer that does basic round-robin load balancing. At Netflix, a much more sophisticated load balancer wraps Eureka to provide weighted load balancing based on several factors like traffic, resource usage, error conditions etc to provide superior resiliency.

Eureka是服务注册和发现的开源框架

Eureka的优势

  • 提供了完整的服务注册和服务查找功能
  • SpringBoot的紧密结合

Eureka架构

eureka_architecture

eureka的基本架构由三个角色组成:

  1. Eureka Server

    • 提供服务注册和发现

  2. Service Provider

    • 服务提供方
    • 将自身服务注册到Eureka,从而使服务消费方能够找到

  3. Service Consumer

    • 服务消费方
    • 从Eureka获取注册服务列表, 从而能够消费服务

architecture-detail

  • ServiceProvider会向Eureka Server做Register(服务注册)、Renewal(服务续约)、Cancellation(服务下线)等操作
  • Eureka Server之间会做注册服务的同步，从而保证状态的同一
  • Service Consumer会向Eureka Server获取服务列表，并消费服务

服务注册

我们Application使用@EnableDiscoveryClient来进行注册，并且在application.properties中指定服务注册中心的位置

1
2
3
4
5
6
7
8
9
10
11
12
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Import(EnableDiscoveryClientImportSelector.class)
public @interface EnableDiscoveryClient {

	/**
	 * If true, the ServiceRegistry will automatically register the local server.
	 */
	boolean autoRegister() default true;
}

该注解用来开启DiscoveryClient实例

eureka_uml_01

从上图中可以看出服务发现的结构图，左侧是netflix的框架实现部分。

服务列表

我们先找到eureka server url列表配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
/**
    * Get the list of all eureka service urls from properties file for the eureka client to talk to.
    *
    * @param clientConfig the clientConfig to use
    * @param instanceZone The zone in which the client resides
    * @param preferSameZone true if we have to prefer the same zone as the client, false otherwise
    * @return an (ordered) map of zone -> list of urls mappings, with the preferred zone first in iteration order
    */
public static Map<String, List<String>> getServiceUrlsMapFromConfig(EurekaClientConfig clientConfig, String instanceZone, boolean preferSameZone) {
    Map<String, List<String>> orderedUrls = new LinkedHashMap<>();
    String region = getRegion(clientConfig);
    String[] availZones = clientConfig.getAvailabilityZones(clientConfig.getRegion());
    if (availZones == null || availZones.length == 0) {
        availZones = new String[1];
        availZones[0] = DEFAULT_ZONE;
    }
    logger.debug("The availability zone for the given region {} are {}", region, availZones);
    int myZoneOffset = getZoneOffset(instanceZone, preferSameZone, availZones);

    String zone = availZones[myZoneOffset];
    List<String> serviceUrls = clientConfig.getEurekaServerServiceUrls(zone);
    if (serviceUrls != null) {
        orderedUrls.put(zone, serviceUrls);
    }
    int currentOffset = myZoneOffset == (availZones.length - 1) ? 0 : (myZoneOffset + 1);
    while (currentOffset != myZoneOffset) {
        zone = availZones[currentOffset];
        serviceUrls = clientConfig.getEurekaServerServiceUrls(zone);
        if (serviceUrls != null) {
            orderedUrls.put(zone, serviceUrls);
        }
        if (currentOffset == (availZones.length - 1)) {
            currentOffset = 0;
        } else {
            currentOffset++;
        }
    }

    if (orderedUrls.size() < 1) {
        throw new IllegalArgumentException("DiscoveryClient: invalid serviceUrl specified!");
    }
    return orderedUrls;
}

通过region获取zone，默认为default zone。根据zone获取service列表。

默认使用defaultZone。也可以使用eureka.client.region来配置zone

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Override
public List<String> getEurekaServerServiceUrls(String myZone) {
    String serviceUrls = configInstance.getStringProperty(
            namespace + CONFIG_EUREKA_SERVER_SERVICE_URL_PREFIX + "." + myZone, null).get();
    if (serviceUrls == null || serviceUrls.isEmpty()) {
        serviceUrls = configInstance.getStringProperty(
                namespace + CONFIG_EUREKA_SERVER_SERVICE_URL_PREFIX + ".default", null).get();

    }
    if (serviceUrls != null) {
        return Arrays.asList(serviceUrls.split(","));
    }

    return new ArrayList<String>();
}

先从指定的zone获取服务列表，如果为空，获取默认服务列表

服务注册

在DiscoveryClient的构造方法中可以找到各种初始化

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
private void initScheduledTasks() {
    if (clientConfig.shouldFetchRegistry()) {
        // registry cache refresh timer
        int registryFetchIntervalSeconds = clientConfig.getRegistryFetchIntervalSeconds();
        int expBackOffBound = clientConfig.getCacheRefreshExecutorExponentialBackOffBound();
        scheduler.schedule(
                new TimedSupervisorTask(
                        "cacheRefresh",
                        scheduler,
                        cacheRefreshExecutor,
                        registryFetchIntervalSeconds,
                        TimeUnit.SECONDS,
                        expBackOffBound,
                        new CacheRefreshThread()
                ),
                registryFetchIntervalSeconds, TimeUnit.SECONDS);
    }

    if (clientConfig.shouldRegisterWithEureka()) {
        int renewalIntervalInSecs = instanceInfo.getLeaseInfo().getRenewalIntervalInSecs();
        int expBackOffBound = clientConfig.getHeartbeatExecutorExponentialBackOffBound();
        logger.info("Starting heartbeat executor: " + "renew interval is: {}", renewalIntervalInSecs);

        // Heartbeat timer
        scheduler.schedule(
                new TimedSupervisorTask(
                        "heartbeat",
                        scheduler,
                        heartbeatExecutor,
                        renewalIntervalInSecs,
                        TimeUnit.SECONDS,
                        expBackOffBound,
                        new HeartbeatThread()
                ),
                renewalIntervalInSecs, TimeUnit.SECONDS);

        // InstanceInfo replicator
        instanceInfoReplicator = new InstanceInfoReplicator(
                this,
                instanceInfo,
                clientConfig.getInstanceInfoReplicationIntervalSeconds(),
                2); // burstSize

        statusChangeListener = new ApplicationInfoManager.StatusChangeListener() {
            @Override
            public String getId() {
                return "statusChangeListener";
            }

            @Override
            public void notify(StatusChangeEvent statusChangeEvent) {
                if (InstanceStatus.DOWN == statusChangeEvent.getStatus() ||
                        InstanceStatus.DOWN == statusChangeEvent.getPreviousStatus()) {
                    // log at warn level if DOWN was involved
                    logger.warn("Saw local status change event {}", statusChangeEvent);
                } else {
                    logger.info("Saw local status change event {}", statusChangeEvent);
                }
                instanceInfoReplicator.onDemandUpdate();
            }
        };

        if (clientConfig.shouldOnDemandUpdateStatusChange()) {
            applicationInfoManager.registerStatusChangeListener(statusChangeListener);
        }

        instanceInfoReplicator.start(clientConfig.getInitialInstanceInfoReplicationIntervalSeconds());
    } else {
        logger.info("Not registering with Eureka server per configuration");
    }
}

如果需要获取注册信息,使用定时器刷新注册缓存。如果要注册服务，定时器定时发送心跳来续约。创建服务信息实例，其会执行定时任务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
InstanceInfoReplicator
public void run() {
    try {
        discoveryClient.refreshInstanceInfo();

        Long dirtyTimestamp = instanceInfo.isDirtyWithTime();
        if (dirtyTimestamp != null) {
            discoveryClient.register();
            instanceInfo.unsetIsDirty(dirtyTimestamp);
        }
    } catch (Throwable t) {
        logger.warn("There was a problem with the instance info replicator", t);
    } finally {
        Future next = scheduler.schedule(this, replicationIntervalSeconds, TimeUnit.SECONDS);
        scheduledPeriodicRef.set(next);
    }
}

刷新实例信息。更新服务注册，如果注册时间过期，重新注册

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/**
    * Register with the eureka service by making the appropriate REST call.
    */
boolean register() throws Throwable {
    logger.info(PREFIX + "{}: registering service...", appPathIdentifier);
    EurekaHttpResponse<Void> httpResponse;
    try {
        httpResponse = eurekaTransport.registrationClient.register(instanceInfo);
    } catch (Exception e) {
        logger.warn(PREFIX + "{} - registration failed {}", appPathIdentifier, e.getMessage(), e);
        throw e;
    }
    if (logger.isInfoEnabled()) {
        logger.info(PREFIX + "{} - registration status: {}", appPathIdentifier, httpResponse.getStatusCode());
    }
    return httpResponse.getStatusCode() == 204;
}

使用EurekaHttpClient注册

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Override
public EurekaHttpResponse<Void> register(InstanceInfo info) {
    String urlPath = "apps/" + info.getAppName();
    ClientResponse response = null;
    try {
        Builder resourceBuilder = jerseyClient.resource(serviceUrl).path(urlPath).getRequestBuilder();
        addExtraHeaders(resourceBuilder);
        response = resourceBuilder
                .header("Accept-Encoding", "gzip")
                .type(MediaType.APPLICATION_JSON_TYPE)
                .accept(MediaType.APPLICATION_JSON)
                .post(ClientResponse.class, info);
        return anEurekaHttpResponse(response.getStatus()).headers(headersOf(response)).build();
    } finally {
        if (logger.isDebugEnabled()) {
            logger.debug("Jersey HTTP POST {}/{} with instance {}; statusCode={}", serviceUrl, urlPath, info.getId(),
                    response == null ? "N/A" : response.getStatus());
        }
        if (response != null) {
            response.close();
        }
    }
}

使用post请求完成服务注册

服务获取
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class CacheRefreshThread implements Runnable {
    public void run() {
        refreshRegistry();
    }
}

void refreshRegistry() {
    try {
            boolean isFetchingRemoteRegionRegistries = isFetchingRemoteRegionRegistries();

            boolean remoteRegionsModified = false;
            // This makes sure that a dynamic change to remote regions to fetch is honored.
            String latestRemoteRegions = clientConfig.fetchRegistryForRemoteRegions();
            if (null != latestRemoteRegions) {
                String currentRemoteRegions = remoteRegionsToFetch.get();
                if (!latestRemoteRegions.equals(currentRemoteRegions)) {
                    // Both remoteRegionsToFetch and AzToRegionMapper.regionsToFetch need to be in sync
                    synchronized (instanceRegionChecker.getAzToRegionMapper()) {
                        if (remoteRegionsToFetch.compareAndSet(currentRemoteRegions, latestRemoteRegions)) {
                            String[] remoteRegions = latestRemoteRegions.split(",");
                            remoteRegionsRef.set(remoteRegions);
                            instanceRegionChecker.getAzToRegionMapper().setRegionsToFetch(remoteRegions);
                            remoteRegionsModified = true;
                        } else {
                            logger.info("Remote regions to fetch modified concurrently," +
                                    " ignoring change from {} to {}", currentRemoteRegions, latestRemoteRegions);
                        }
                    }
                } else {
                    // Just refresh mapping to reflect any DNS/Property change
                    instanceRegionChecker.getAzToRegionMapper().refreshMapping();
                }
            }

            boolean success = fetchRegistry(remoteRegionsModified);
            if (success) {
                registrySize = localRegionApps.get().size();
                lastSuccessfulRegistryFetchTimestamp = System.currentTimeMillis();
            }
    }
}

获取最新的region，然后根据最新的region获取最新的注册列表
服务续约
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 /**
    * Renew with the eureka service by making the appropriate REST call
    */
boolean renew() {
    EurekaHttpResponse<InstanceInfo> httpResponse;
    try {
        httpResponse = eurekaTransport.registrationClient.sendHeartBeat(instanceInfo.getAppName(), instanceInfo.getId(), instanceInfo, null);
        logger.debug(PREFIX + "{} - Heartbeat status: {}", appPathIdentifier, httpResponse.getStatusCode());
        if (httpResponse.getStatusCode() == 404) {
            REREGISTER_COUNTER.increment();
            logger.info(PREFIX + "{} - Re-registering apps/{}", appPathIdentifier, instanceInfo.getAppName());
            long timestamp = instanceInfo.setIsDirtyWithTime();
            boolean success = register();
            if (success) {
                instanceInfo.unsetIsDirty(timestamp);
            }
            return success;
        }
        return httpResponse.getStatusCode() == 200;
    } catch (Throwable e) {
        logger.error(PREFIX + "{} - was unable to send heartbeat!", appPathIdentifier, e);
        return false;
    }
}

发送POST心跳来续约

服务注册中心

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
@POST
@Consumes({"application/json", "application/xml"})
public Response addInstance(InstanceInfo info,
                            @HeaderParam(PeerEurekaNode.HEADER_REPLICATION) String isReplication) {
    logger.debug("Registering instance {} (replication={})", info.getId(), isReplication);
    // validate that the instanceinfo contains all the necessary required fields
    if (isBlank(info.getId())) {
        return Response.status(400).entity("Missing instanceId").build();
    } else if (isBlank(info.getHostName())) {
        return Response.status(400).entity("Missing hostname").build();
    } else if (isBlank(info.getIPAddr())) {
        return Response.status(400).entity("Missing ip address").build();
    } else if (isBlank(info.getAppName())) {
        return Response.status(400).entity("Missing appName").build();
    } else if (!appName.equals(info.getAppName())) {
        return Response.status(400).entity("Mismatched appName, expecting " + appName + " but was " + info.getAppName()).build();
    } else if (info.getDataCenterInfo() == null) {
        return Response.status(400).entity("Missing dataCenterInfo").build();
    } else if (info.getDataCenterInfo().getName() == null) {
        return Response.status(400).entity("Missing dataCenterInfo Name").build();
    }

    // handle cases where clients may be registering with bad DataCenterInfo with missing data
    DataCenterInfo dataCenterInfo = info.getDataCenterInfo();
    if (dataCenterInfo instanceof UniqueIdentifier) {
        String dataCenterInfoId = ((UniqueIdentifier) dataCenterInfo).getId();
        if (isBlank(dataCenterInfoId)) {
            boolean experimental = "true".equalsIgnoreCase(serverConfig.getExperimental("registration.validation.dataCenterInfoId"));
            if (experimental) {
                String entity = "DataCenterInfo of type " + dataCenterInfo.getClass() + " must contain a valid id";
                return Response.status(400).entity(entity).build();
            } else if (dataCenterInfo instanceof AmazonInfo) {
                AmazonInfo amazonInfo = (AmazonInfo) dataCenterInfo;
                String effectiveId = amazonInfo.get(AmazonInfo.MetaDataKey.instanceId);
                if (effectiveId == null) {
                    amazonInfo.getMetadata().put(AmazonInfo.MetaDataKey.instanceId.getName(), info.getId());
                }
            } else {
                logger.warn("Registering DataCenterInfo of type {} without an appropriate id", dataCenterInfo.getClass());
            }
        }
    }

    registry.register(info, "true".equals(isReplication));
    return Response.status(204).build();  // 204 to be backwards compatible
}

对注册信息进行校验,然后使用PeerAwareInstanceRegistry来进行注册

1
2
3
4
5
6
7
8
9
@Override
public void register(final InstanceInfo info, final boolean isReplication) {
    int leaseDuration = Lease.DEFAULT_DURATION_IN_SECS;
    if (info.getLeaseInfo() != null && info.getLeaseInfo().getDurationInSecs() > 0) {
        leaseDuration = info.getLeaseInfo().getDurationInSecs();
    }
    super.register(info, leaseDuration, isReplication);
    replicateToPeers(Action.Register, info.getAppName(), info.getId(), info, null, isReplication);
}

检查租赁时间，注册指定时长的服务。同步信息到各个服务节点

总结

整个eureka的服务注册和发现内部都是用REST方式来进行通讯以达到信息同步的目的

Posted on

简介

Spring Cloud是一个基于Spring Boot实现的云应用开发工具，它为基于JVM的云应用开发中涉及的配置管理、服务发现、断路器、智能路由、微代理、控制总线、全局锁、决策竞选、分布式会话和集群状态管理等操作提供了一种简单的开发方式。

微服务架构

微服务架构就是将一个完整的应用从数据存储开始垂直拆分成多个不同的服务，每个服务都能独立部署、独立维护、独立扩展，服务与服务间通过诸如RESTful API的方式互相调用。可以参考Martin Fower的《MicroServices》

架构

springcloud_archtecture

技术组成

SpringCloudTechs

  • 服务治理：这是Spring Cloud的核心。目前Spring Cloud主要通过整合Netflix的相关产品来实现这方面的功能（Spring Cloud Netflix），包括用于服务注册和发现的Eureka，调用断路器Hystrix，调用端负载均衡Ribbon，Rest客户端Feign，智能服务路由Zuul，用于监控数据收集和展示的Spectator、Servo、Atlas，用于配置读取的Archaius和提供Controller层Reactive封装的RxJava。

对于服务的注册和发现，除了Eureka，Spring Cloud也整合了Consul和Zookeeper作为备选，但是因为这两个方案在CAP理论上都遵循CP而不是AP，所以官方并没有推荐使用。

  • 分布式链路监控：Spring Cloud Sleuth提供了全自动、可配置的数据埋点，以收集微服务调用链路上的性能数据，并发送给Zipkin进行存储、统计和展示。

  • 消息组件：Spring Cloud Stream对于分布式消息的各种需求进行了抽象，包括发布订阅、分组消费、消息分片等功能，实现了微服务之间的异步通信。Spring Cloud Stream也集成了第三方的RabbitMQ和Apache Kafka作为消息队列的实现。而Spring Cloud Bus基于Spring Cloud Stream，主要提供了服务间的事件通信（比如刷新配置）。

  • 配置中心：基于Spring Cloud Netflix和Spring Cloud Bus，Spring又提供了Spring Cloud Config，实现了配置集中管理、动态刷新的配置中心概念。配置通过Git或者简单文件来存储，支持加解密。

  • 安全控制：Spring Cloud Security基于OAUTH2这个开放网络的安全标准，提供了微服务环境下的单点登录、资源授权、令牌管理等功能。

  • 命令行工具：Spring Cloud Cli提供了以命令行和脚本的方式来管理微服务及Spring Cloud组件的方式。

  • 集群工具：Spring Cloud Cluster提供了集群选主、分布式锁、一次性令牌等分布式集群需要的技术组件。

Posted on

ActivityManagerService可以认为是Android系统四大组件的通讯中心。四大组件相关的操作都会跟它打交道。而PackageManagerService则主要负责应用包内容的解析校验。因此在插件化过程中，Hook它们可以带来很多方便

AMS

1
ActivityManager.getService();

通过上述代码，我们能够获得AMS

1
2
3
4
5
6
7
8
9
10
11
12
13
public static IActivityManager getService() {
        return IActivityManagerSingleton.get();
    }

private static final Singleton<IActivityManager> IActivityManagerSingleton =
        new Singleton<IActivityManager>() {
            @Override
            protected IActivityManager create() {
                final IBinder b = ServiceManager.getService(Context.ACTIVITY_SERVICE);
                final IActivityManager am = IActivityManager.Stub.asInterface(b);
                return am;
            }
        };

其最终通过ServiceManager获取对应的IBinder对象，然后转成对应的接口，也就是AMS

Hook AMS

通过查看源码，我们能够知道在8.0及之后,ActivityManagerNative以及标记过期，即将被去掉。而是直接使用ActivityManager来获取服务。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class AmsPmsHookHelperKt {
    companion object {
        fun hookActivityManager() {
            try {
                val activityManagerNativeClass = Class.forName("android.app.ActivityManagerNative")

                val gDefaultField = activityManagerNativeClass.getDeclaredField("gDefault")
                gDefaultField.isAccessible = true
                val gDefault = gDefaultField.get(null)

                val singletonClass = Class.forName("android.util.Singleton")
                val instanceField = singletonClass.getDeclaredField("mInstance")
                instanceField.isAccessible = true

                val rawIActivityManager = instanceField.get(gDefault)
                val activityManagerInterface = Class.forName("android.app.IActivityManager")
                val proxy = Proxy.newProxyInstance(Thread.currentThread().contextClassLoader, arrayOf(activityManagerInterface), HookHandler(rawIActivityManager))
                instanceField.set(gDefault, proxy)
            } catch (e: Exception) {
                e.printStackTrace()
            }
        }
    }
}

获取ActivityManagerNative，通过动态代理，重新创建出一个对象，用来替换gDefault

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class HookHandler implements InvocationHandler {

    private static final String TAG = "HookHandler";

    private Object mBase;

    public HookHandler(Object base) {
        mBase = base;
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        Log.d(TAG, "hey, baby; you are hooked!!");
        Log.d(TAG, "method:" + method.getName() + " called with args:" + Arrays.toString(args));

        return method.invoke(mBase, args);
    }
}

当IActivityManager触发调用时，invoke回调，这个时候会打印出添加的内容

1
2
10-04 16:04:13.092 17046-17046/cn.binea.pluginframeworkdemo D/HookHandler: hey, baby; you are hooked!!
10-04 16:04:13.093 17046-17046/cn.binea.pluginframeworkdemo D/HookHandler: method:activityIdle called with args:[android.os.BinderProxy@e0086ab, {1.0 310mcc260mnc [en_US] ldltr sw360dp w360dp h568dp 480dpi nrml port finger qwerty/v/v -nav/h s.6}, false]

PMS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@Override
public PackageManager getPackageManager() {
    if (mPackageManager != null) {
        return mPackageManager;
    }

    IPackageManager pm = ActivityThread.getPackageManager();
    if (pm != null) {
        // Doesn't matter if we make more than one instance.
        return (mPackageManager = new ApplicationPackageManager(this, pm));
    }

    return null;
}

ContextImpl中通过ActivityThread.getPackageManager获取pms，并会通过ApplicationPackageManager增强pms

1
2
3
4
5
6
7
8
9
10
11
public static IPackageManager getPackageManager() {
    if (sPackageManager != null) {
        //Slog.v("PackageManager", "returning cur default = " + sPackageManager);
        return sPackageManager;
    }
    IBinder b = ServiceManager.getService("package");
    //Slog.v("PackageManager", "default service binder = " + b);
    sPackageManager = IPackageManager.Stub.asInterface(b);
    //Slog.v("PackageManager", "default service = " + sPackageManager);
    return sPackageManager;
}

ThreadActivity中，检查缓存pms是否存在。如果不存在，通过ServiceManager获取特定IBinder对象，并通过asInterface转换成对应的接口IPackageManager.这里可以Hook这个缓存对象。

Posted on

Binder是Android系统的通讯桥梁。因此Hook Binder就能够改变通讯行为.分析Hook过程有利于理解Binder机制

系统服务获取

1
ActivityManager am = (ActivityManager)context.getSystemService(Context.ACTIVITY_SERVICE)

通过以上代码能够获取系统服务。我们来看看内部是如何获取的

1
2
3
4
5
6
7
8
9
@Override
public Object getSystemService(String name) {
    return SystemServiceRegistry.getSystemService(this, name);
}

@Override
public String getSystemServiceName(Class<?> serviceClass) {
    return SystemServiceRegistry.getSystemServiceName(serviceClass);
}

ContextImpl中通过SystemServiceRegistry获取服务

1
2
3
4
5
6
7
8
public static Object getSystemService(ContextImpl ctx, String name) {
    ServiceFetcher<?> fetcher = SYSTEM_SERVICE_FETCHERS.get(name);
    return fetcher != null ? fetcher.getService(ctx) : null;
}

public static String getSystemServiceName(Class<?> serviceClass) {
    return SYSTEM_SERVICE_NAMES.get(serviceClass);
}

通过Class或者类限定名都可以获取对应的服务

1
2
3
4
private static final HashMap<Class<?>, String> SYSTEM_SERVICE_NAMES =
            new HashMap<Class<?>, String>();
private static final HashMap<String, ServiceFetcher<?>> SYSTEM_SERVICE_FETCHERS =
        new HashMap<String, ServiceFetcher<?>>();

这里有两个HashMap，一个用于缓存服务名称，一个用于缓存服务获取器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
static {
    registerService(Context.ACTIVITY_SERVICE, ActivityManager.class,
            new CachedServiceFetcher<ActivityManager>() {
        @Override
        public ActivityManager createService(ContextImpl ctx) {
            return new ActivityManager(ctx.getOuterContext(), ctx.mMainThread.getHandler());
        }});
}

private static <T> void registerService(String serviceName, Class<T> serviceClass,
        ServiceFetcher<T> serviceFetcher) {
    SYSTEM_SERVICE_NAMES.put(serviceClass, serviceName);
    SYSTEM_SERVICE_FETCHERS.put(serviceName, serviceFetcher);
}

服务在首次使用的时候已经被初始化了。并且初始化时候会被缓存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
static abstract class CachedServiceFetcher<T> implements ServiceFetcher<T> {
    private final int mCacheIndex;

    public CachedServiceFetcher() {
        mCacheIndex = sServiceCacheSize++;
    }

    @Override
    @SuppressWarnings("unchecked")
    public final T getService(ContextImpl ctx) {
        final Object[] cache = ctx.mServiceCache;
        synchronized (cache) {
            // Fetch or create the service.
            Object service = cache[mCacheIndex];
            if (service == null) {
                try {
                    service = createService(ctx);
                    cache[mCacheIndex] = service;
                } catch (ServiceNotFoundException e) {
                    onServiceNotFound(e);
                }
            }
            return (T)service;
        }
    }

    public abstract T createService(ContextImpl ctx) throws ServiceNotFoundException;
}

每次CacheServiceFetcher创建时，全局的服务缓存空间会增大，并将新的服务放到数组末端。每次需要获取服务的时候，先寻找缓存中的服务，如果不存在创建并缓存

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
private static final Singleton<IActivityManager> IActivityManagerSingleton =
    new Singleton<IActivityManager>() {
        @Override
        protected IActivityManager create() {
            final IBinder b = ServiceManager.getService(Context.ACTIVITY_SERVICE);
            final IActivityManager am = IActivityManager.Stub.asInterface(b);
            return am;
        }
    };

public static IBinder getService(String name) {
    try {
        IBinder service = sCache.get(name);
        if (service != null) {
            return service;
        } else {
            return Binder.allowBlocking(getIServiceManager().getService(name));
        }
    } catch (RemoteException e) {
        Log.e(TAG, "error in getService", e);
    }
    return null;
}

在ActivityManager中可以看到ActivityManagerService服务的获取方式，从ServiceManager中获取。在ServiceManager中，会先从缓冲中获取。

Hook IBinder

从上述流程中可以了解系统服务的获取过程，那么我们需要怎么去Hook这些服务呢？我们可以将asInterface返回的结果修改成我们Hook过的对象。下面我们以ClipboardService为例来实践Hook过程

1
2
3
4
5
public ClipboardManager(Context context, Handler handler) throws ServiceNotFoundException {
    mContext = context;
    mService = IClipboard.Stub.asInterface(
            ServiceManager.getServiceOrThrow(Context.CLIPBOARD_SERVICE));
}

ClipboardManager在初始化的时候，从ServiceManager获取ClipboardService。从这里可以看出asInterface需要一个IBinder参数，这个参数从ServiceManager获取。而在ServiceManager中会首先从缓存中查找。因此我们可以提前往缓存冲插入我们Hook的对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
@SuppressLint("PrivateApi")
class BinderHookHelperKt {
    companion object {

        @Throws(Exception::class)
        fun hookClipboardService() {
            val CLIPBOARD_SERVICE = "clipboard"
            val serviceManager = Class.forName("android.os.ServiceManager")
            val getService = serviceManager.getDeclaredMethod("getService", String::class.java)
            val rawBinder = getService.invoke(null, CLIPBOARD_SERVICE) as IBinder
            val hookedBinder = Proxy.newProxyInstance(serviceManager.classLoader, arrayOf(IBinder::class.java), BinderProxyHookHandlerKt(rawBinder))
            val cacheField = serviceManager.getDeclaredField("sCache")
            cacheField.isAccessible = true
            val cache = cacheField.get(null) as HashMap<String, IBinder>
            cache.put(CLIPBOARD_SERVICE, hookedBinder as IBinder)
        }
    }
}

获取ServiceManager的getService方法，返回一个IBinder对象。然后通过动态代理，代理此IBinder对象。然后将插入ServiceManager的缓存中。这样下次使用时就能够直接拿到这个代理对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class BinderProxyHookHandlerKt(val base: IBinder) : InvocationHandler {

    companion object {
        val TAG = BinderProxyHookHandlerKt::class.java.canonicalName
    }

    var stub = Any()
    var iinterface = Any()

    init {
        try {
            stub = Class.forName("android.content.IClipboard\$Stub")
            iinterface = Class.forName("android.content.IClipboard")
        } catch (e: Exception) {
            e.printStackTrace()
        }
    }

    override fun invoke(proxy: Any?, method: Method?, args: Array<out Any>?): Any {
        if ("queryLocalInterface" == method!!.name) {
            Log.d(TAG, "hook queryLocalInterface")
            val classArray = arrayOf(iinterface as Class<Any>)
            return Proxy.newProxyInstance((proxy as Any).javaClass.classLoader, classArray, BinderHookHandlerKt(base, stub as Class<Any>))
        }

        Log.d(TAG, "method: " + method.name)
        return method.invoke(base, args)
    }
}

如果IBinder的queryLocalInterface触发，那么通过动态代理，代理IClipboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
class BinderHookHandlerKt(base: IBinder, stubClass: Class<Any>) : InvocationHandler {
    companion object {
        val TAG = BinderHookHandlerKt::class.java.canonicalName
    }

    var obj = Any()

    init {
        try {
            val asInterfaceMethod: Method = stubClass.getDeclaredMethod("asInterface", IBinder::class.java)
            obj = asInterfaceMethod.invoke(null, base)
        } catch (e: Exception) {
            throw RuntimeException("hooked failed")
        }
    }

    override fun invoke(proxy: Any?, method: Method?, args: Array<out Any>?): Any {
        if ("getPrimaryClip" == method!!.name) {
            Log.d(TAG, "hook getPrimaryClip")
            return ClipData.newPlainText(null, "you have been hooked")
        }

        if ("hasPrimaryClip" == method.name) {
            return true
        }

        return method.invoke(obj, args)
    }
}

获取IClipboard$Stub的asInterface方法，返回BinderProxy。当getPrimaryClip方法触发时，返回固定值。当hasPrimaryClip触发时，总是返回true表示粘贴板有内容。这样就成功的hook ClipboardService

运行程序之后，使用粘贴功能，会发现永远只能粘贴我们自己返回的内容

整个过程的思路是:
ServiceManager内部一张表管理着很多的Binder对象。我们需要Hook某个Binder对象的queryLocalInterface，并将其缓存。由于ServiceManager的缓存表里的IBinder大部分都是BinderProxy对象。当使用时会调用asInterface转换成需要的接口。

Posted on

使用动态代理能够达到AOP编程的效果。现在流行的插件化框架也广泛使用了动态代理。本文主要讲解动态代理的Hook机制

代理

代理可以实现方法增强，主要分为静态代理和动态代理。

静态代理

所有的原始类，代理类都提供好了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
//目标对象接口
interface DoSth {
    fun doSomething(sth: Long): Array<Any>
}

//目标对象实现
class DoSthImpl : DoSth {

    override fun doSomething(sth: Long): Array<Any> {
        println("DoSthImpl " + sth)
        return arrayOf("a", "b", "c");
    }
}

//代理对象实现
class ProxyDoSth(val base: DoSth) : DoSth {

    override fun doSomething(sth: Long): Array<Any> {
        println("ProxyDoSth " + (sth * 10))
        val sth = base.doSomething(sth)
        sth[0] = "d"
        return sth
    }
}

//测试程序
val sthImpl = DoSthImpl()
val proxyDoSth = ProxyDoSth(sthImpl)
println(Arrays.toString(proxyDoSth.doSomething(10)))

代理对象实现将目标对象的实现行为改变了，将传入的值扩大10倍，并将返回的数组内容改变。因此代理模式是可以改变目标对象的行为的

动态代理

静态代理使用简单，但是比较繁琐。当需要代理的类较多时，会很麻烦。JDK提供了动态代理，运行时生成对应的代理类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
//代理方法
class InvocationHandlerImpl(val base: Any) : InvocationHandler {
    override fun invoke(p0: Any?, p1: Method?, p2: Array<out Any>?): Any {

        if ("doSomething" == (p1!!.name)) {
            val value: Long = p2!![0] as Long
            val doSthValue = value * 5
            println(doSthValue)
            val invoke: Array<Any> = p1.invoke(base, doSthValue) as Array<Any>
            invoke[0] = "d"
            return invoke
        }

        return Unit
    }
}

//测试程序
val sthImpl = DoSthImpl()
val result = Proxy.newProxyInstance(DoSth::class.java.classLoader, sthImpl.javaClass.interfaces, InvocationHandlerImpl(sthImpl))
val ds = (result as DoSth).doSomething(10)

代理模式，运行时生成对应代理接口的代理实现类，当目标方法调用时，触发Invocation.invoke.

代理Hook

Hook之前，得先找到Hook点，一般是静态变量或者单例，因为这些对象不会经常变化。下面我们来分析如何Hook Activity。都知道Activity的启动都是通过Instrumentation。而其ActivityThread内部，一个进程只有一个ActivityThread。因此只要Hook住这个对象那么就可以操作Instrumentation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@SuppressLint("PrivateApi")
class HookHelper {
    companion object {

        @Throws(Exception::class)
        fun attachContext() {
            val activityThreadClass = Class.forName("android.app.ActivityThread")
            val currentActivityThreadMethod = activityThreadClass.getDeclaredMethod("currentActivityThread")
            currentActivityThreadMethod.isAccessible = true
            val currentActivityThread = currentActivityThreadMethod.invoke(null)
            val instrumentationField = activityThreadClass.getDeclaredField("mInstrumentation")
            instrumentationField.isAccessible = true
            val instrumentation = instrumentationField.get(currentActivityThread) as Instrumentation

            //代理instrumentation
            val proxyInstrumentation = ProxyInstrumentation(instrumentation)
            instrumentationField.set(currentActivityThread, proxyInstrumentation)
        }
    }
}

HookHelper通过反射获取当前进程的ActivityThread，然后替换内部的Instrumentation字段

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class ProxyInstrumentation(val instrumentation: Instrumentation) : Instrumentation() {
    companion object {
        val TAG = ProxyInstrumentation::class.java.canonicalName
    }

    fun execStartActivity(
            who: Context, contextThread: IBinder?, token: IBinder?, target: Activity?,
            intent: Intent, requestCode: Int, options: Bundle?): Instrumentation.ActivityResult? {
        Log.d(TAG, "\nstartActivity, who = [" + who + "], " +
                "\ncontextThread = [" + contextThread + "], \ntoken = [" + token + "], " +
                "\ntarget = [" + target + "], \nintent = [" + intent +
                "], \nrequestCode = [" + requestCode + "], \noptions = [" + options + "]")

        try {
            val execStartActivity = Instrumentation::class.java.getDeclaredMethod("execStartActivity", Context::class.java, IBinder::class.java, IBinder::class.java, Activity::class.java, Intent::class.java, Int::class.javaPrimitiveType, Bundle::class.java)
            execStartActivity.isAccessible = true
            return execStartActivity.invoke(instrumentation, who, contextThread, token, target, intent, requestCode, options) as Instrumentation.ActivityResult?
        } catch (e: Exception) {
            throw RuntimeException("don't support hook")
        }

    }
}

ProxyInstrumentation继承自Instrumentation，用于增强Instrumentation。在每个Activity启动之前打印日志

在App启动的时候修改替换Instrumentation

1
2
3
4
5
6
7
8
9
10
class MyApp : Application() {
    override fun attachBaseContext(base: Context?) {
        super.attachBaseContext(base)
        try {
            HookHelper.attachContext()
        } catch (e: Exception) {
            e.printStackTrace()
        }
    }
}

这个时候启动App，然后重新启动一个Activity,那么会打印日志

1
startActivity, who = [cn.binea.pluginframeworkdemo.MyApp@3cc96c5], contextThread = [android.app.ActivityThread$ApplicationThread@56a681a], token = [null], target = [null], intent = [Intent { act=android.intent.action.VIEW dat=http://www.baidu.com/... flg=0x10000000 }], requestCode = [-1], options = [null]

至此，这个例子简单的Hook了Activity的启动，使其有了别的功能。

Posted on

BroadcastReceiver

一般广播的使用，都是先注册，后发送。注册可以分为静态注册和动态注册

动态注册

ContextImpl.registerReceiver
1
2
3
4
5
6
@Override
    public Intent registerReceiver(BroadcastReceiver receiver, IntentFilter filter,
            String broadcastPermission, Handler scheduler) {
        return registerReceiverInternal(receiver, getUserId(),
                filter, broadcastPermission, scheduler, getOuterContext(), 0);
}
ContextImpl.registerReceiverInternal
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
private Intent registerReceiverInternal(BroadcastReceiver receiver, int userId,
            IntentFilter filter, String broadcastPermission,
            Handler scheduler, Context context, int flags) {
    IIntentReceiver rd = null;
    if (receiver != null) {
        if (mPackageInfo != null && context != null) {
            if (scheduler == null) {
                scheduler = mMainThread.getHandler();
            }
            rd = mPackageInfo.getReceiverDispatcher(
                receiver, context, scheduler,
                mMainThread.getInstrumentation(), true);
        } else {
            if (scheduler == null) {
                scheduler = mMainThread.getHandler();
            }
            rd = new LoadedApk.ReceiverDispatcher(
                    receiver, context, scheduler, null, true).getIIntentReceiver();
        }
    }
    try {
        final Intent intent = ActivityManager.getService().registerReceiver(
                mMainThread.getApplicationThread(), mBasePackageName, rd, filter,
                broadcastPermission, userId, flags);
        if (intent != null) {
            intent.setExtrasClassLoader(getClassLoader());
            intent.prepareToEnterProcess();
        }
        return intent;
    } catch (RemoteException e) {
        throw e.rethrowFromSystemServer();
    }
}

通过ContextImpl注册。创建IIntentReceiver，然后通过ActivityManagerService注册广播

AMS.registerReceiver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
public Intent registerReceiver(IApplicationThread caller, String callerPackage,
            IIntentReceiver receiver, IntentFilter filter, String permission, int userId,
            int flags) {
                ...
            ReceiverList rl = mRegisteredReceivers.get(receiver.asBinder());
            if (rl == null) {
                rl = new ReceiverList(this, callerApp, callingPid, callingUid,
                        userId, receiver);
                if (rl.app != null) {
                    rl.app.receivers.add(rl);
                } else {
                    try {
                        receiver.asBinder().linkToDeath(rl, 0);
                    } catch (RemoteException e) {
                        return sticky;
                    }
                    rl.linkedToDeath = true;
                }
                mRegisteredReceivers.put(receiver.asBinder(), rl);
            }

            ...
            final int stickyCount = allSticky.size();
            for (int i = 0; i < stickyCount; i++) {
                Intent intent = allSticky.get(i);
                BroadcastQueue queue = broadcastQueueForIntent(intent);
                BroadcastRecord r = new BroadcastRecord(queue, intent, null,
                        null, -1, -1, false, null, null, AppOpsManager.OP_NONE, null, receivers,
                        null, 0, null, null, false, true, true, -1);
                queue.enqueueParallelBroadcastLocked(r);
                queue.scheduleBroadcastsLocked();
            }
}

检查是否是sticky broadcast。搜索已注册的广播，如果不存在，创建并缓存。通过每个广播意图，获取对应的前台广播队列或者后台广播队列。然后将新的广播信息入列

BroadcastQueue.scheduleBroadcastsLocked
1
2
3
4
5
6
7
8
9
10
11
public void scheduleBroadcastsLocked() {
    if (DEBUG_BROADCAST) Slog.v(TAG_BROADCAST, "Schedule broadcasts ["
            + mQueueName + "]: current="
            + mBroadcastsScheduled);

    if (mBroadcastsScheduled) {
        return;
    }
    mHandler.sendMessage(mHandler.obtainMessage(BROADCAST_INTENT_MSG, this));
    mBroadcastsScheduled = true;
}

通过Handler来处理广播消息

BroadcastQueue.processNextBroadcast
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
final void processNextBroadcast(boolean fromMsg) {
    ...
    while (mParallelBroadcasts.size() > 0) {
        r = mParallelBroadcasts.remove(0);
        ...
        final int N = r.receivers.size();
        for (int i=0; i<N; i++) {
            Object target = r.receivers.get(i);
            if (DEBUG_BROADCAST)  Slog.v(TAG_BROADCAST,
                    "Delivering non-ordered on [" + mQueueName + "] to registered "
                    + target + ": " + r);
            deliverToRegisteredReceiverLocked(r, (BroadcastFilter)target, false, i);
        }
        addBroadcastToHistoryLocked(r);
    }

    ...
    performReceiveLocked(r.callerApp, r.resultTo,
                                new Intent(r.intent), r.resultCode,
                                r.resultData, r.resultExtras, false, false, r.userId);
}

首先处理所有无序广播，然后处理有序广播。之后调用performReceiveLocked

BroadcastQueue.performReceiveLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
void performReceiveLocked(ProcessRecord app, IIntentReceiver receiver,
            Intent intent, int resultCode, String data, Bundle extras,
            boolean ordered, boolean sticky, int sendingUser) throws RemoteException {
    // Send the intent to the receiver asynchronously using one-way binder calls.
    if (app != null) {
        if (app.thread != null) {
            // If we have an app thread, do the call through that so it is
            // correctly ordered with other one-way calls.
            try {
                app.thread.scheduleRegisteredReceiver(receiver, intent, resultCode,
                        data, extras, ordered, sticky, sendingUser, app.repProcState);
            // TODO: Uncomment this when (b/28322359) is fixed and we aren't getting
            // DeadObjectException when the process isn't actually dead.
            //} catch (DeadObjectException ex) {
            // Failed to call into the process.  It's dying so just let it die and move on.
            //    throw ex;
            } catch (RemoteException ex) {
                // Failed to call into the process. It's either dying or wedged. Kill it gently.
                synchronized (mService) {
                    Slog.w(TAG, "Can't deliver broadcast to " + app.processName
                            + " (pid " + app.pid + "). Crashing it.");
                    app.scheduleCrash("can't deliver broadcast");
                }
                throw ex;
            }
        } else {
            // Application has died. Receiver doesn't exist.
            throw new RemoteException("app.thread must not be null");
        }
    } else {
        receiver.performReceive(intent, resultCode, data, extras, ordered,
                sticky, sendingUser);
    }
}

如果app进程还存在，那么直接通知ApplicationThread处理消息，以保证广播的顺序。最终都是调用IIntentReceiver

LoadApk.ReceiverDispatcher.InnerReceiver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
@Override
public void performReceive(Intent intent, int resultCode, String data,
        Bundle extras, boolean ordered, boolean sticky, int sendingUser) {
    final LoadedApk.ReceiverDispatcher rd;
    if (intent == null) {
        Log.wtf(TAG, "Null intent received");
        rd = null;
    } else {
        rd = mDispatcher.get();
    }
    if (ActivityThread.DEBUG_BROADCAST) {
        int seq = intent.getIntExtra("seq", -1);
        Slog.i(ActivityThread.TAG, "Receiving broadcast " + intent.getAction()
                + " seq=" + seq + " to " + (rd != null ? rd.mReceiver : null));
    }
    if (rd != null) {
        rd.performReceive(intent, resultCode, data, extras,
                ordered, sticky, sendingUser);
    } else {
        // The activity manager dispatched a broadcast to a registered
        // receiver in this process, but before it could be delivered the
        // receiver was unregistered.  Acknowledge the broadcast on its
        // behalf so that the system's broadcast sequence can continue.
        if (ActivityThread.DEBUG_BROADCAST) Slog.i(ActivityThread.TAG,
                "Finishing broadcast to unregistered receiver");
        IActivityManager mgr = ActivityManager.getService();
        try {
            if (extras != null) {
                extras.setAllowFds(false);
            }
            mgr.finishReceiver(this, resultCode, data, extras, false, intent.getFlags());
        } catch (RemoteException e) {
            throw e.rethrowFromSystemServer();
        }
    }
}

通过IPC从AMS通讯回到LoadedApk

静态注册

在AndroidManifest.xml中注册广播。广播发出后，会经由PackageManagerService获取当前进程的所有配置信息，然后会做比对

广播发送

AMS.broadcastIntent
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public final int broadcastIntent(IApplicationThread caller,
            Intent intent, String resolvedType, IIntentReceiver resultTo,
            int resultCode, String resultData, Bundle resultExtras,
            String[] requiredPermissions, int appOp, Bundle bOptions,
            boolean serialized, boolean sticky, int userId) {
        synchronized(this) {
            intent = verifyBroadcastLocked(intent);
            ...
            int res = broadcastIntentLocked(callerApp,
                    callerApp != null ? callerApp.info.packageName : null,
                    intent, resolvedType, resultTo, resultCode, resultData, resultExtras,
                    requiredPermissions, appOp, bOptions, serialized, sticky,
                    callingPid, callingUid, userId);
        }
}

校验广播意图

AMS.broadcastIntentLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
final int broadcastIntentLocked(ProcessRecord callerApp,
            String callerPackage, Intent intent, String resolvedType,
            IIntentReceiver resultTo, int resultCode, String resultData,
            Bundle resultExtras, String[] requiredPermissions, int appOp, Bundle bOptions,
            boolean ordered, boolean sticky, int callingPid, int callingUid, int userId) {
                ...

                if (!ordered && NR > 0) {
            // If we are not serializing this broadcast, then send the
            // registered receivers separately so they don't wait for the
            // components to be launched.
            if (isCallerSystem) {
                checkBroadcastFromSystem(intent, callerApp, callerPackage, callingUid,
                        isProtectedBroadcast, registeredReceivers);
            }
            final BroadcastQueue queue = broadcastQueueForIntent(intent);
            BroadcastRecord r = new BroadcastRecord(queue, intent, callerApp,
                    callerPackage, callingPid, callingUid, callerInstantApp, resolvedType,
                    requiredPermissions, appOp, brOptions, registeredReceivers, resultTo,
                    resultCode, resultData, resultExtras, ordered, sticky, false, userId);
            if (DEBUG_BROADCAST) Slog.v(TAG_BROADCAST, "Enqueueing parallel broadcast " + r);
            final boolean replaced = replacePending
                    && (queue.replaceParallelBroadcastLocked(r) != null);
            // Note: We assume resultTo is null for non-ordered broadcasts.
            if (!replaced) {
                queue.enqueueParallelBroadcastLocked(r);
                queue.scheduleBroadcastsLocked();
            }
            registeredReceivers = null;
            NR = 0;
        }
}

首先处理系统的广播消息。然后将广播消息入队列

BroadCastQueue.processNextBroadcast
1
2
3
4
5
6
7
8
9
10
11
final void processNextBroadcast(boolean fromMsg) {
    ...
    if (app != null && app.thread != null && !app.killed) {
        try {
            app.addPackage(info.activityInfo.packageName,
                    info.activityInfo.applicationInfo.versionCode, mService.mProcessStats);
            processCurBroadcastLocked(r, app);
            return;
        }
    }
}

如果当前进程还在，那么处理当前广播

BroadCastQueue.processCurBroadcastLocked
1
2
3
4
5
6
7
8
private final void processCurBroadcastLocked(BroadcastRecord r,
            ProcessRecord app) throws RemoteException {
                ...
                app.thread.scheduleReceiver(new Intent(r.intent), r.curReceiver,
                    mService.compatibilityInfoForPackageLocked(r.curReceiver.applicationInfo),
                    r.resultCode, r.resultData, r.resultExtras, r.ordered, r.userId,
                    app.repProcState);
}

如果当前进程正在备份，那么忽略广播。更新进程状态，调整优先级。然后通知ApplicationThread处理广播

ActivityThread.handleReceiver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
private void handleReceiver(ReceiverData data) {
    // If we are getting ready to gc after going to the background, well
    // we are back active so skip it.
    unscheduleGcIdler();

    String component = data.intent.getComponent().getClassName();

    LoadedApk packageInfo = getPackageInfoNoCheck(
            data.info.applicationInfo, data.compatInfo);

    IActivityManager mgr = ActivityManager.getService();

    Application app;
    BroadcastReceiver receiver;
    ContextImpl context;
    try {
        app = packageInfo.makeApplication(false, mInstrumentation);
        context = (ContextImpl) app.getBaseContext();
        if (data.info.splitName != null) {
            context = (ContextImpl) context.createContextForSplit(data.info.splitName);
        }
        java.lang.ClassLoader cl = context.getClassLoader();
        data.intent.setExtrasClassLoader(cl);
        data.intent.prepareToEnterProcess();
        data.setExtrasClassLoader(cl);
        receiver = (BroadcastReceiver)cl.loadClass(component).newInstance();
    } catch (Exception e) {
        if (DEBUG_BROADCAST) Slog.i(TAG,
                "Finishing failed broadcast to " + data.intent.getComponent());
        data.sendFinished(mgr);
        throw new RuntimeException(
            "Unable to instantiate receiver " + component
            + ": " + e.toString(), e);
    }

    try {
        if (localLOGV) Slog.v(
            TAG, "Performing receive of " + data.intent
            + ": app=" + app
            + ", appName=" + app.getPackageName()
            + ", pkg=" + packageInfo.getPackageName()
            + ", comp=" + data.intent.getComponent().toShortString()
            + ", dir=" + packageInfo.getAppDir());

        sCurrentBroadcastIntent.set(data.intent);
        receiver.setPendingResult(data);
        receiver.onReceive(context.getReceiverRestrictedContext(),
                data.intent);
    } catch (Exception e) {
        if (DEBUG_BROADCAST) Slog.i(TAG,
                "Finishing failed broadcast to " + data.intent.getComponent());
        data.sendFinished(mgr);
        if (!mInstrumentation.onException(receiver, e)) {
            throw new RuntimeException(
                "Unable to start receiver " + component
                + ": " + e.toString(), e);
        }
    } finally {
        sCurrentBroadcastIntent.set(null);
    }

    if (receiver.getPendingResult() != null) {
        data.finish();
    }
}

通过反射，创建广播对象，然后调用onReceive

时序图

注册

register

发送

send

Posted on

App启动点

ActivityThread.performLaunchActivity

一个APP是从桌面开始启动。其实也是启动一个新的ACTIVITY的过程。因此主要关注ActivityThread.performLaunchActivity

1
2
3
try {
    Application app = r.packageInfo.makeApplication(false, mInstrumentation);
}

能够看出这个时候去创建了一个Application。因此LoadApk是apk的启动点

LoadApk.makeApplication

1
2
3
4
5
6
7
if (mApplication != null) {
            return mApplication;
}
...
ContextImpl appContext = ContextImpl.createAppContext(mActivityThread, this);
app = mActivityThread.mInstrumentation.newApplication(
        cl, appClass, appContext);
  • 如果application已经存在，那么直接返回
  • 否则会创建ContextImpl，然后通过Instrumentation反射创建出Application

LoadApk的由来

ActivityRecordClient.packageInfo

在源码里可以看出LoadApk是在H Handler中创建的

1
2
3
4
5
6
7
8
case LAUNCH_ACTIVITY: {
    Trace.traceBegin(Trace.TRACE_TAG_ACTIVITY_MANAGER, "activityStart");
    final ActivityClientRecord r = (ActivityClientRecord) msg.obj;

    r.packageInfo = getPackageInfoNoCheck(
            r.activityInfo.applicationInfo, r.compatInfo);
    handleLaunchActivity(r, null, "LAUNCH_ACTIVITY");
}
  • 获取LoadApk

ActivityThread.getPackageInfo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
private LoadedApk getPackageInfo(ApplicationInfo aInfo, CompatibilityInfo compatInfo,
            ClassLoader baseLoader, boolean securityViolation, boolean includeCode,
            boolean registerPackage) {
                WeakReference<LoadedApk> ref;
            if (differentUser) {
                // Caching not supported across users
                ref = null;
            } else if (includeCode) {
                ref = mPackages.get(aInfo.packageName);
            } else {
                ref = mResourcePackages.get(aInfo.packageName);
            }

            ...
            //创建新的LoadApk
    packageInfo =
            new LoadedApk(this, aInfo, compatInfo, baseLoader,
                    securityViolation, includeCode &&
                    (aInfo.flags&ApplicationInfo.FLAG_HAS_CODE) != 0, registerPackage);
}
  • 如果app已经启动，那么按照包名从缓存中获取。这里的缓存都是弱引用
  • 如果是一个未启动的app，那么会创建新的LoadApk。然后放入缓存

在LoadedApk中，能看到其用了系统classLoader

1
mClassLoader = ClassLoader.getSystemClassLoader();

ClassLoader

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
private static ClassLoader createSystemClassLoader() {
    String classPath = System.getProperty("java.class.path", ".");
    String librarySearchPath = System.getProperty("java.library.path", "");

    // String[] paths = classPath.split(":");
    // URL[] urls = new URL[paths.length];
    // for (int i = 0; i < paths.length; i++) {
    // try {
    // urls[i] = new URL("file://" + paths[i]);
    // }
    // catch (Exception ex) {
    // ex.printStackTrace();
    // }
    // }
    //
    // return new java.net.URLClassLoader(urls, null);

    // TODO Make this a java.net.URLClassLoader once we have those?
    return new PathClassLoader(classPath, librarySearchPath, BootClassLoader.getInstance());
    }
  • 最终会根据classPath,libraryPath以及BootClassLoader来创建PathClassLoader。其首个参数代表的就是apk中的classes.dex的路径
1
2
3
4
5
public BaseDexClassLoader(String dexPath, File optimizedDirectory,
            String librarySearchPath, ClassLoader parent) {
        super(parent);
        this.pathList = new DexPathList(this, dexPath, librarySearchPath, null);
}
  • 这里会创建DexPathList
1
2
this.dexElements = makeDexElements(splitDexPath(dexPath), optimizedDirectory,suppressedExceptions, definingContext);

DexPathList.makeDexElements

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
private static Element[] makeDexElements(List<File> files, File optimizedDirectory,
            List<IOException> suppressedExceptions, ClassLoader loader) {
    Element[] elements = new Element[files.size()];
    int elementsPos = 0;
    /*
    * Open all files and load the (direct or contained) dex files up front.
    */
    for (File file : files) {
        if (file.isDirectory()) {
            // We support directories for looking up resources. Looking up resources in
            // directories is useful for running libcore tests.
            elements[elementsPos++] = new Element(file);
        } else if (file.isFile()) {
            String name = file.getName();

            if (name.endsWith(DEX_SUFFIX)) {
                // Raw dex file (not inside a zip/jar).
                try {
                    DexFile dex = loadDexFile(file, optimizedDirectory, loader, elements);
                    if (dex != null) {
                        elements[elementsPos++] = new Element(dex, null);
                    }
                } catch (IOException suppressed) {
                    System.logE("Unable to load dex file: " + file, suppressed);
                    suppressedExceptions.add(suppressed);
                }
            } else {
                DexFile dex = null;
                try {
                    dex = loadDexFile(file, optimizedDirectory, loader, elements);
                } catch (IOException suppressed) {
                    /*
                    * IOException might get thrown "legitimately" by the DexFile constructor if
                    * the zip file turns out to be resource-only (that is, no classes.dex file
                    * in it).
                    * Let dex == null and hang on to the exception to add to the tea-leaves for
                    * when findClass returns null.
                    */
                    suppressedExceptions.add(suppressed);
                }

                if (dex == null) {
                    elements[elementsPos++] = new Element(file);
                } else {
                    elements[elementsPos++] = new Element(dex, file);
                }
            }
        } else {
            System.logW("ClassLoader referenced unknown path: " + file);
        }
    }
    if (elementsPos != elements.length) {
        elements = Arrays.copyOf(elements, elementsPos);
    }
    return elements;
}
  • 遍历所有文件，如果是目录，将目录存到Element数组中
    8 如果是.dex文件，那么尝试加载，然后存在Element数组中。因此最终.dex的信息都会在Element数组中，那么如果替换dex，那么原理上也是可以的。很多热修复框架就是基于这个原理
  • 上述这个是不分包的情况

MultiDexApplication

1
2
3
4
5
6
7
public class MultiDexApplication extends Application {
  @Override
  protected void attachBaseContext(Context base) {
    super.attachBaseContext(base);
    MultiDex.install(this);
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
...
String apkPath = applicationInfo.sourceDir;
if (installedApk.contains(apkPath)) {
    return;
}
installedApk.add(apkPath);
...
clearOldDexDir(context);

File dexDir = getDexDir(context, applicationInfo);
List<? extends File> files =
    MultiDexExtractor.load(context, applicationInfo, dexDir, false);
installSecondaryDexes(loader, dexDir, files);
  • apk若已经安装过，那么直接返回
  • 清除旧的dex目录
  • 获取Dex目录
  • 加载dex文件
1
2
3
4
5
6
7
8
9
static List<? extends File> load(Context context, ApplicationInfo applicationInfo, File dexDir,
            boolean forceReload) throws IOException {
                ...
                //如果没有修改过，那么加载之前存在的dex
                files = loadExistin
                gExtractions(context, sourceApk, dexDir);
                ...
                //解压新的dex
}
  • 加载旧的
1
2
3
4
5
6
7
private static List<ExtractedDex> loadExistingExtractions(
            Context context, File sourceApk, File dexDir)
            throws IOException {
                ...for (int secondaryNumber = 2; secondaryNumber <= totalDexNumber; secondaryNumber++) {
                    ...
                }
}

  • 依次加载从2开始的classes

  • 加载新的

1
2
3
4
5
private static List<ExtractedDex> performExtractions(File sourceApk, File dexDir)
            throws IOException {
                ...
                 prepareDexDir(dexDir, extractedFilePrefix);
            }
  • 首先移除老的dex
  • 依次从classes2.dex加载文件

MultiDex.installSecondaryDexes

1
2
3
4
5
6
7
8
9
if (!files.isEmpty()) {
    if (Build.VERSION.SDK_INT >= 19) {
        V19.install(loader, files, dexDir);
    } else if (Build.VERSION.SDK_INT >= 14) {
        V14.install(loader, files, dexDir);
    } else {
        V4.install(loader, files);
    }
}

按系统版本区分加载

v19以上

1
2
3
4
5
6
7
8
9
10
private static void install(ClassLoader loader,
                List<? extends File> additionalClassPathEntries,
                File optimizedDirectory)
                ...
                Field pathListField = findField(loader, "pathList");
                ...
                expandFieldArray(dexPathList, "dexElements", makeDexElements(dexPathList,
                    new ArrayList<File>(additionalClassPathEntries), optimizedDirectory,
                    suppressedExceptions));
}
  • 通过反射找到pathList
  • 将加载出来的dex数组合并到原有的dexElements后面

v14 跟 v19一样，只是少了很多异常处理

v4 合并path, files, zips, dex。之后的版本有Element来存储这些信息

热修复可以基于这种dex的原理，来加载自己的dex文件。

Posted on Edited on

本文基于aosp8.0源码分析

Service的启动

Service的启动分两种，一种startService，另一种bindService

StartService

一般我们会在Activity中通过startService来启动一个服务

Activity.startService

其内部实际上是由ContextWrapper的startService来启动，实际启动的地方是在ContextImpl.startService

ContextImpl.startService
1
2
3
4
5
@Override
public ComponentName startService(Intent service) {
    warnIfCallingFromSystemProcess();
    return startServiceCommon(service, false, mUser);
}
ContextImpl.startServiceCommon
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
private ComponentName startServiceCommon(Intent service, boolean requireForeground,
        UserHandle user) {
    try {
        validateServiceIntent(service);
        service.prepareToLeaveProcess(this);
        ComponentName cn = ActivityManager.getService().startService(
            mMainThread.getApplicationThread(), service, service.resolveTypeIfNeeded(
                        getContentResolver()), requireForeground,
                        getOpPackageName(), user.getIdentifier());
        if (cn != null) {
            if (cn.getPackageName().equals("!")) {
                throw new SecurityException(
                        "Not allowed to start service " + service
                        + " without permission " + cn.getClassName());
            } else if (cn.getPackageName().equals("!!")) {
                throw new SecurityException(
                        "Unable to start service " + service
                        + ": " + cn.getClassName());
            } else if (cn.getPackageName().equals("?")) {
                throw new IllegalStateException(
                        "Not allowed to start service " + service + ": " + cn.getClassName());
            }
        }
        return cn;
    } catch (RemoteException e) {
        throw e.rethrowFromSystemServer();
    }
}
  • 这里首选需要验证服务,5.0之后隐世服务就不被允许，何为隐世，未指定包名，未指定类名
1
2
3
public static IActivityManager getService() {
    return IActivityManagerSingleton.get();
}
  • 从ServiceManager中获取ActivityManager的IBinder对象
  • 通过IPC获取了ActivityManagerService对象
  • 每个系统服务都采用了AIDL的方式进行，不直接对外暴露代理
IActivityManager.aidl

该文件对应的native实现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
class BpActivityManager : public BpInterface<IActivityManager>
{
public:
    explicit BpActivityManager(const sp<IBinder>& impl)
        : BpInterface<IActivityManager>(impl)
    {
    }

    virtual int openContentUri(const String16& stringUri)
    {
        Parcel data, reply;
        data.writeInterfaceToken(IActivityManager::getInterfaceDescriptor());
        data.writeString16(stringUri);
        status_t ret = remote()->transact(OPEN_CONTENT_URI_TRANSACTION, data, & reply);
        int fd = -1;
        if (ret == NO_ERROR) {
            int32_t exceptionCode = reply.readExceptionCode();
            if (!exceptionCode) {
                // Success is indicated here by a nonzero int followed by the fd;
                // failure by a zero int with no data following.
                if (reply.readInt32() != 0) {
                    fd = fcntl(reply.readParcelFileDescriptor(), F_DUPFD_CLOEXEC, 0);
                }
            } else {
                // An exception was thrown back; fall through to return failure
                ALOGD("openContentUri(%s) caught exception %d\n",
                        String8(stringUri).string(), exceptionCode);
            }
        }
        return fd;
    }
};
ActivityManagerService.startService
1
2
3
4
5
6
7
8
9
10
11
12
public class ActivityManagerService extends IActivityManager.Stub {
@Override
    public ComponentName startService(IApplicationThread caller, Intent service,
            String resolvedType, boolean requireForeground, String callingPackage, int userId)
            throws TransactionTooLargeException {
                ...
                res = mServices.startServiceLocked(caller, service,
                        resolvedType, callingPid, callingUid,
                        requireForeground, callingPackage, userId);
                ...
    }
}
  • ActivityManagerService就是ActivityManager定义的AIDL的具体实现
  • 调用ActiveServices.startServiceLocked
ActiveServices.startServiceLocked
1
2
3
4
5
6
7
8
ComponentName startServiceLocked(IApplicationThread caller, Intent service, String resolvedType,
            int callingPid, int callingUid, boolean fgRequired, String callingPackage, final int userId)
            throws TransactionTooLargeException {
                ...
                //启动服务
                ComponentName cmp = startServiceInnerLocked(smap, service, r, callerFg, addToStarting);
                ...
}

检查Service调用者合法性。创建并缓存Service信息

ActiveServices.startServiceInnerLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
ComponentName startServiceInnerLocked(ServiceMap smap, Intent service, ServiceRecord r,
            boolean callerFg, boolean addToStarting) throws TransactionTooLargeException {
    ServiceState stracker = r.getTracker();
    if (stracker != null) {
        stracker.setStarted(true, mAm.mProcessStats.getMemFactorLocked(), r.lastActivity);
    }
    r.callStart = false;
    synchronized (r.stats.getBatteryStats()) {
        r.stats.startRunningLocked();
    }
    String error = bringUpServiceLocked(r, service.getFlags(), callerFg, false, false);
    if (error != null) {
        return new ComponentName("!!", error);
    }

    if (r.startRequested && addToStarting) {
        boolean first = smap.mStartingBackground.size() == 0;
        smap.mStartingBackground.add(r);
        r.startingBgTimeout = SystemClock.uptimeMillis() + mAm.mConstants.BG_START_TIMEOUT;
        if (DEBUG_DELAYED_SERVICE) {
            RuntimeException here = new RuntimeException("here");
            here.fillInStackTrace();
            Slog.v(TAG_SERVICE, "Starting background (first=" + first + "): " + r, here);
        } else if (DEBUG_DELAYED_STARTS) {
            Slog.v(TAG_SERVICE, "Starting background (first=" + first + "): " + r);
        }
        if (first) {
            smap.rescheduleDelayedStartsLocked();
        }
    } else if (callerFg || r.fgRequired) {
        smap.ensureNotStartingBackgroundLocked(r);
    }

    return r.name;
}

调用bringUpServiceLocked，处理返回结果

ActiveService.bringUpServiceLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
private String bringUpServiceLocked(ServiceRecord r, int intentFlags, boolean execInFg,
            boolean whileRestarting, boolean permissionsReviewRequired)
            throws TransactionTooLargeException {
    if (r.app != null && r.app.thread != null) {
        //服务已经启动
        sendServiceArgsLocked(r, execInFg, false);
        return null;
    }

    ...
    app = mAm.getProcessRecordLocked(procName, r.appInfo.uid, false);
    if (DEBUG_MU) Slog.v(TAG_MU, "bringUpServiceLocked: appInfo.uid=" + r.appInfo.uid
                + " app=" + app);
    if (app != null && app.thread != null) {
        try {
            app.addPackage(r.appInfo.packageName, r.appInfo.versionCode, mAm.mProcessStats);
            realStartServiceLocked(r, app, execInFg);
            return null;
        } catch (TransactionTooLargeException e) {
            throw e;
        } catch (RemoteException e) {
            Slog.w(TAG, "Exception when starting service " + r.shortName, e);
        }

        // If a dead object exception was thrown -- fall through to
        // restart the application.
    }
    ...
}

如果服务已经启动，调用sendserviceArgsLocked。首次创建。获取进程信息。调用realStartServiceLocked

ActiveServices.realStartServiceLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
private final void realStartServiceLocked(ServiceRecord r,
            ProcessRecord app, boolean execInFg) throws RemoteException {
                ...
                final boolean newService = app.services.add(r);
                ...
                ...
                app.thread.scheduleCreateService(r, r.serviceInfo,
                    mAm.compatibilityInfoForPackageLocked(r.serviceInfo.applicationInfo),
                    app.repProcState);
                ...

                sendServiceArgsLocked(r, execInFg, true);
}

更新服务优先级。通知ApplicationThread创建启动Service。之后调用sendServiceArgsLocked，其作用，处理Service参数，并回调onStartCommand

ActivityThread.scheduleCreateService
1
2
3
4
5
6
7
8
9
10
public final void scheduleCreateService(IBinder token,
                ServiceInfo info, CompatibilityInfo compatInfo, int processState) {
    updateProcessState(processState, false);
    CreateServiceData s = new CreateServiceData();
    s.token = token;
    s.info = info;
    s.compatInfo = compatInfo;

    sendMessage(H.CREATE_SERVICE, s);
}

  • H Handler发出消息，创建service

  • 总结; 在8.0中，一个服务的启动流程是:
    ContextImpl.startService -> startCommonService->ActivityManager.getService->ServiceManager.getService->(IPC) AMS.startService-> ActiveService.startServiceLocked->startServiceInnerLocked->bringUpServiceLocked->realStartServiceLocked->ApplicationThread.scheduleCreateService->H.handleCreateService

BindService

bindService的流程，跟startService很相似。

ActiveServices.bindServiceLocked
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
int bindServiceLocked(IApplicationThread caller, IBinder token, Intent service,
            String resolvedType, final IServiceConnection connection, int flags,
            String callingPackage, final int userId) throws TransactionTooLargeException {
                ...
                ArrayList<ConnectionRecord> clist = s.connections.get(binder);
            if (clist == null) {
                clist = new ArrayList<ConnectionRecord>();
                s.connections.put(binder, clist);
            }
            clist.add(c);
            b.connections.add(c);
            if (activity != null) {
                if (activity.connections == null) {
                    activity.connections = new HashSet<ConnectionRecord>();
                }
                activity.connections.add(c);
            }
            b.client.connections.add(c);
            ...

            if ((flags&Context.BIND_AUTO_CREATE) != 0) {
                s.lastActivity = SystemClock.uptimeMillis();
                if (bringUpServiceLocked(s, service.getFlags(), callerFg, false,
                        permissionsReviewRequired) != null) {
                    return 0;
                }
            }
            ...
}

ServiceRecord存储每个Binder，也就是ServiceConnection。内部的数据结构都是Set，确保Binder唯一。调用bringUpServiceLocked

ActiveServices.bringUpServiceLocked
1
2
3
4
5
private String bringUpServiceLocked(ServiceRecord r, int intentFlags, boolean execInFg,
            boolean whileRestarting, boolean permissionsReviewRequired)
            throws TransactionTooLargeException {

}

首次绑定Service的过程和startService一样。也是调用realStartServiceLocked。非首次也是调用sendServiceArgsLocked,这个函数中，如果有startService的话，那么会发出消息通知ApplicationThread需要处理onStartCommand。BindService下，会直接返回

1
2
3
4
5
6
7
8
private final void realStartServiceLocked(ServiceRecord r,
            ProcessRecord app, boolean execInFg) throws RemoteException {
    ...
    //创建过程一样

    ...
    requestServiceBindingsLocked(r, execInFg);
}

创建过程一样。创建完之后会检查是否有Binder。如果有直接通知ApplicationThread需要绑定服务。

总结

客户端发起启动服务请求,通过IPC的方式通知SystemServer进程中的ActivityManagerService，然后通知ApplicationThread创建服务

时序图

StartService

startService

BindService

bindService

Posted on

更新:

2017-09-09
2016-10-02

引言

本文主要分析OkHttp3的原理。。其有很多特点:

  • HTTP2/SPDY支持所有面向相同主机的请求共享一个Socket
  • 连接池减少请求时间，减少握手次数
  • 基于Headers的缓存策略
  • 响应缓存避免重复的请求
  • 线程队列，支持高效并发

本文基于OkHttp3.10.0-SNAPSHOT，需要弄懂一下几个问题

  • 连接池如何维护
  • 线程队列如何维护
  • 缓存机制如何实现
  • 重试机制是怎样的

用法

同步
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
OkHttpClient client = new OkHttpClient();

//get
Request request = new Request.Builder()
    .url(url)
    .build();

//post
RequestBody body = ReqeustBody.create(JSON, json);
Request request = new Request.Builder()
    .url(url)
    .build();
    
try (Response response = client.newCall(request).execute()) {
  return response.body().string();
}

异步
1
2
3
4
5
6
7
8
9
10
11
12
13
14
OkHttpClient client = new OkHttpClient();

//get
Request request = new Request.Builder()
    .url(url)
    .build();

//post
RequestBody body = ReqeustBody.create(JSON, json);
Request request = new Request.Builder()
    .url(url)
    .build();
    
client.newCall(request).enqueue(Callback)

源码分析

OkHttpClient

OkHttpClient创建的时候，如果没有执行建造器，会使用默认构造器。这里用到了建造者模式，便于参数的配置。默认参数包括

  • 支持协议: HTTP/2,HTTP1.1，默认没有支持(SPDY)
  • 传输层协议: TLS(传输层安全协议SNI,ALPN)，未加密协议
  • 代理选择器: 使用系统默认的代理选择器
  • Socket工厂: 默认
  • 主机名验证: 包括验证IP,域名及证书验证X509认证
  • 证书信任验证: 防止证书认证攻击和中间人攻击
  • 连接池
  • dns
  • 各种超时(连接，读，写: 10s
1
2
3
@Override public Call newCall(Request request) {
    return new RealCall(this, request, false /* for web socket */);
}
  • 创建RealCall
RealCall
1
2
3
4
5
6
private RealCall(OkHttpClient client, Request originalRequest, boolean forWebSocket) {
    this.client = client;
    this.originalRequest = originalRequest;
    this.forWebSocket = forWebSocket;
    this.retryAndFollowUpInterceptor = new RetryAndFollowUpInterceptor(client, forWebSocket);
  }
  • 创建RealCall时会创建RetryAndFollowUpInterceptor

请求创建完成，接下来开始发起请求

同步请求execute
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
@Override public Response execute() throws IOException {
    synchronized (this) {
      if (executed) throw new IllegalStateException("Already Executed");
      executed = true;
    }
    captureCallStackTrace();
    eventListener.callStart(this);
    try {
      //通过Dispatcher执行请求
      client.dispatcher().executed(this);
      Response result = getResponseWithInterceptorChain();
      if (result == null) throw new IOException("Canceled");
      return result;
    } catch (IOException e) {
      eventListener.callFailed(this, e);
      throw e;
    } finally {
      client.dispatcher().finished(this);
    }
}
  • 通过Dispatcher下发请求。这里是同步的请求。
1
2
3
synchronized void executed(RealCall call) {
    runningSyncCalls.add(call);
}
  • Dispatcher内部有个存储同步运行的Deque双端队列，队列默认大小是16
1
private final Deque<RealCall> runningSyncCalls = new ArrayDeque<>();

对于ArrayDeque来说,其内部如果容量不够，每次都会翻倍扩容，当大小超过Integer.MAX_VALUE时会抛出异常

1
2
3
int newCapacity = n << 1;
if (newCapacity < 0)
    throw new IllegalStateException("Sorry, deque too big");

因此到这里，也仅仅只是将请求放入到双端队列中，还没真正开始执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Response getResponseWithInterceptorChain() throws IOException {
    // Build a full stack of interceptors.
    List<Interceptor> interceptors = new ArrayList<>();
    //添加自定义的interceptor
    interceptors.addAll(client.interceptors());
    //添加重试机制
    interceptors.add(retryAndFollowUpInterceptor);
    //桥接用户请求到网络请求
    interceptors.add(new BridgeInterceptor(client.cookieJar()));
    //加入缓存机制
    interceptors.add(new CacheInterceptor(client.internalCache()));
    //加入连接拦截器
    interceptors.add(new ConnectInterceptor(client));
    if (!forWebSocket) {
      //非websocket,网络拦截器
      interceptors.addAll(client.networkInterceptors());
    }
    //服务器请求拦截器
    interceptors.add(new CallServerInterceptor(forWebSocket));

    //拦截链开始执行
    Interceptor.Chain chain = new RealInterceptorChain(interceptors, null, null, null, 0,
        originalRequest, this, eventListener, client.connectTimeoutMillis(),
        client.readTimeoutMillis(), client.writeTimeoutMillis());

    return chain.proceed(originalRequest);
  }
  • 添加拦截器，默认有重试拦截器，桥接拦截器，缓存拦截器，连接拦截器，网络拦截器，调用服务器拦截器。这里可以看得出，将网络请求细化成了很多小部分，便于对每部分的控制。然后开始执行请求。
拦截器链(RealInterceptorChain)
1
2
3
4
5
6
7
8
9
10
public Response proceed(Request request, StreamAllocation streamAllocation, HttpCodec httpCodec,
      RealConnection connection) throws IOException {
        ...
        RealInterceptorChain next = new RealInterceptorChain(interceptors, streamAllocation, httpCodec,
        connection, index + 1, request, call, eventListener, connectTimeout, readTimeout,
        writeTimeout);
        Interceptor interceptor = interceptors.get(index);
    Response response = interceptor.intercept(next);
        ...
}
  • 检查是否已经编码，如果已经编过码，检查连接的url是否正确
  • 创建新的连接链，并传入下一个拦截器。直到所有的拦截器执行完成，返回结果。
Dispatcher异步请求策略
1
2
3
4
5
6
7
8
synchronized void enqueue(AsyncCall call) {
    if (runningAsyncCalls.size() < maxRequests && runningCallsForHost(call) < maxRequestsPerHost) {
      runningAsyncCalls.add(call);
      executorService().execute(call);
    } else {
      readyAsyncCalls.add(call);
    }
}
  • 主要由两个双端队列来维护，一个正在运行请求的队列，一个准备好请求的队列
  • 如果队列的大小小于最大请求数并且队列请求中同一个主机的请求数小于最大请求数，添加到队列，否则添加到准备队列。默认最大请求数64，同一主机最大请求数5
  • 使用线程队列执行每个运行中的请求，线程队列采用的SynchronousQueue，控制多线程并发，其内部永远是个空状态，任务插入操作之后，必然等待任务取出操作，内部有个TransferQueue来讲任务插入或取出，确保操作线程安全
AsyncCall

内部的流程跟同步的流程一样

上述不管是同步，或者异步，执行完之后都会调用Dispatcher进行finish操作。来将任务移出队列。如果是异步，还会将准备队列的任务加入到运行队列，并交由线程池运行。至此，一次请求执行完成。这里面拦截器很重要，会在之后详细描述

拦截器(Interceptor)

RetryAndFollowUpInterceptor
1
2
3
4
5
/**
   * How many redirects and auth challenges should we attempt? Chrome follows 21 redirects; Firefox,
   * curl, and wget follow 20; Safari follows 16; and HTTP/1.0 recommends 5.
   */
  private static final int MAX_FOLLOW_UPS = 20;

关于重试次数，Chrome使用21次，Firefox,curl和wget使用20次,Safari使用16次，HTTP/1.0建议5次
最大重试20次

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@Override public Response intercept(Chain chain) throws IOException {
  ...
  streamAllocation = new StreamAllocation(client.connectionPool(), createAddress(request.url()),
        call, eventListener, callStackTrace);

  while (true) {
      if (canceled) {
        streamAllocation.release();
        throw new IOException("Canceled");
      }

      try {
        response = realChain.proceed(request, streamAllocation, null, null);
        releaseConnection = false;
      }catch(RouteException e) {
        recover()
      }catch(IOException e) {
        recover()
      }finally {
        //other exception release
      }

      Request followUp = followUpRequest(response);
  }
  ...
}
  • 创建StreamAllocation协调连接池，地址信息以及请求序列
  • 如果请求取消了，那么释放当前Socket资源
  • 尝试发出网络请求，捕获这个过程的异常。
    • 如果是RouteException，尝试恢复
    • IOException异常
  • 本次请求成功，释放资源
  • 如果之前请求结果存在，获取该响应对应的请求followUpRequest
  • 如果请求不存在，直接返回该响应
  • 如果请求次数超过上限，抛出异常
  • 如果请求体已经编过码，抛出异常
  • 响应和请求不是相同的地址，创建新的StreamAllocation
  • 保存该请求和该响应
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
private boolean recover(IOException e, boolean requestSendStarted, Request userRequest) {
    streamAllocation.streamFailed(e);

    // The application layer has forbidden retries.
    if (!client.retryOnConnectionFailure()) return false;

    // We can't send the request body again.
    if (requestSendStarted && userRequest.body() instanceof UnrepeatableRequestBody) return false;

    // This exception is fatal.
    if (!isRecoverable(e, requestSendStarted)) return false;

    // No more routes to attempt.
    if (!streamAllocation.hasMoreRoutes()) return false;

    // For failure recovery, use the same route selector with a new connection.
    return true;
}
  • 尝试重新连接，判断客户端是否允许重试，无法再次发送请求,没有更多的路由可选择，以及各种协议异常。如果以上情况出现则，不会进行重试连接
1
2
3
4
5
6
7
8
private Request followUpRequest(Response userResponse) throws IOException {
  ...
  int responseCode = userResponse.code();
  switch(responseCode) {
      ...
  }
  ...
}
  • 获取响应码，根据各种响应码，构建出各自的请求，如果响应码是正常的，那么不会返回新的请求体。本次请求成功

因此重试的整体流程是

  • 尝试连接->如果出现(Route/IOException)，判断能否重试->如果之前响应体存在，创造空内容的响应体，根据响应体的响应码，构建出请求，如果请求为空，那么本次请求成功，判断是否超过重试次数，请求能否发出，请求和响应地址是否一致
BridgeInterceptor

桥接客户端代码到网络代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@Override public Response intercept(Chain chain) throws IOException {
  ...
  ///加入头部压缩
  boolean transparentGzip = false;
  if (userRequest.header("Accept-Encoding") == null && userRequest.header("Range") == null) {
    transparentGzip = true;
    requestBuilder.header("Accept-Encoding", "gzip");
  }
  //桥接头部
  ...
  Response networkResponse = chain.proceed(requestBuilder.build());

  ...
  if (transparentGzip
        && "gzip".equalsIgnoreCase(networkResponse.header("Content-Encoding"))
        && HttpHeaders.hasBody(networkResponse)) {
      GzipSource responseBody = new GzipSource(networkResponse.body().source());
      Headers strippedHeaders = networkResponse.headers().newBuilder()
          .removeAll("Content-Encoding")
          .removeAll("Content-Length")
          .build();
      responseBuilder.headers(strippedHeaders);
      String contentType = networkResponse.header("Content-Type");
      responseBuilder.body(new RealResponseBody(contentType, -1L, Okio.buffer(responseBody)));
    }
}
  • 首先桥接请求头
  • 接下来进入缓存拦截器
  • 处理请求体压缩，使用GzipSource
CacheInterceptor
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
@Override public Response intercept(Chain chain) throws IOException {
  Response cacheCandidate = cache != null
        ? cache.get(chain.request())
        : null;
  CacheStrategy strategy = new CacheStrategy.Factory(now, chain.request(), cacheCandidate).get();
  ...
  if (networkRequest == null && cacheResponse == null) {
        return new Response.Builder()
            .request(chain.request())
            .protocol(Protocol.HTTP_1_1)
            .code(504)
            .message("Unsatisfiable Request (only-if-cached)")
            .body(Util.EMPTY_RESPONSE)
            .sentRequestAtMillis(-1L)
            .receivedResponseAtMillis(System.currentTimeMillis())
            .build();
  }

  ...
  networkResponse = chain.proceed(networkRequest);
  ...
  if (cacheResponse != null) {
      if (networkResponse.code() == HTTP_NOT_MODIFIED) {
        cache.trackConditionalCacheHit();
        cache.update(cacheResponse, response);
  }
  ...
  if (cache != null) {
      if (HttpHeaders.hasBody(response) && CacheStrategy.isCacheable(response, networkRequest)) {
        // Offer this request to the cache.
        CacheRequest cacheRequest = cache.put(response);
        return cacheWritingResponse(cacheRequest, response);
      }

      if (HttpMethod.invalidatesCache(networkRequest.method())) {
        try {
          cache.remove(networkRequest);
        } catch (IOException ignored) {
          // The cache cannot be written.
        }
      }
    }
}
  • 获取内部缓存响应，默认使用DiskLruCache
  • 根据响应体的头部信息，创建缓存策略(Date, Expires, Last-Modified, ETag, Age)，记录缓存策略
  • 如果禁止网络请求，并且缓存无法满足，则返回失败信息
  • 禁止网络请求，返回缓存
  • 发起网络请求
  • 既有网络响应又有缓存响应，更新缓存
  • 只有网络请求，没有缓存，如果允许缓存，怎缓存网络响应，无法缓存GET以外的响应.
ConnectInterceptor

向目标服务器打开一个连接

1
2
3
4
5
6
7
8
9
10
@Override public Response intercept(Chain chain) throws IOException {
  ...
  StreamAllocation streamAllocation = realChain.streamAllocation();

    // We need the network to satisfy this request. Possibly for validating a conditional GET.
    boolean doExtensiveHealthChecks = !request.method().equals("GET");
    HttpCodec httpCodec = streamAllocation.newStream(client, chain, doExtensiveHealthChecks);
    RealConnection connection = streamAllocation.connection();
    ...
}
  • 创建HttpCodec
  • 连接服务器
1
2
3
4
5
6
public HttpCodec newStream(
      OkHttpClient client, Interceptor.Chain chain, boolean doExtensiveHealthChecks) {
        ...
        RealConnection resultConnection = findHealthyConnection(connectTimeout, readTimeout,
          writeTimeout, connectionRetryEnabled, doExtensiveHealthChecks);
}
  • 找到可用的连接
  • 将请求编码，Http请求如何映射到Socket,使用HttpCodec

问题,如何找到可用的连连呢？

1
2
3
4
5
6
7
8
private RealConnection findHealthyConnection(int connectTimeout, int readTimeout,
      int writeTimeout, boolean connectionRetryEnabled, boolean doExtensiveHealthChecks) {
         while (true) {
      RealConnection candidate = findConnection(connectTimeout, readTimeout, writeTimeout,
          connectionRetryEnabled);
          ...
    }
}
  • 找出健康连接，何为健康连接,socket没被关闭,socket输入输出关闭,Socket超时,BufferSource耗尽
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
private RealConnection findConnection(int connectTimeout, int readTimeout, int writeTimeout,
      boolean connectionRetryEnabled) throws IOException {
        ...
      if (this.connection != null) {
        // We had an already-allocated connection and it's good.
        result = this.connection;
        releasedConnection = null;
      }  

      if (result == null) {
        // Attempt to get a connection from the pool.
        Internal.instance.get(connectionPool, address, this, null);
        if (connection != null) {
          foundPooledConnection = true;
          result = connection;
        } else {
          selectedRoute = route;
        }
      }
      ...
      for (int i = 0, size = routes.size(); i < size; i++) {
          Route route = routes.get(i);
          Internal.instance.get(connectionPool, address, this, route);
          if (connection != null) {
            foundPooledConnection = true;
            result = connection;
            this.route = route;
            break;
          }
        }
        ...
  result.connect(
        connectTimeout, readTimeout, writeTimeout, connectionRetryEnabled, call, eventListener);
}
  • 如果连接已经存在，可以复用
  • 不存在，根据地址从连接池获取一个连接
  • 选择可用路由
  • 如果没有找到可用连接,接着选择可用路由，并创建连接，连接加入新的请求信息
  • 开始TCP/TLS握手，更新路优库
  • 将连接放入连接池
  • 如果有多个相同地址的连接并发创建了，那么释放当前这个，获取别的

创建HttpCodec

1
2
3
4
5
6
7
8
9
10
11
public HttpCodec newCodec(OkHttpClient client, Interceptor.Chain chain,
      StreamAllocation streamAllocation) throws SocketException {
    if (http2Connection != null) {
      return new Http2Codec(client, chain, streamAllocation, http2Connection);
    } else {
      socket.setSoTimeout(chain.readTimeoutMillis());
      source.timeout().timeout(chain.readTimeoutMillis(), MILLISECONDS);
      sink.timeout().timeout(chain.writeTimeoutMillis(), MILLISECONDS);
      return new Http1Codec(client, streamAllocation, source, sink);
    }
  }
  • 如果Http2不为空，直接返回Http2Codec
  • Socket设置相应的超时，读写时间
CallServerInterceptor

向服务器发起网络请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
@Override public Response intercept(Chain chain) throws IOException {
  ...
  if ("100-continue".equalsIgnoreCase(request.header("Expect"))) {
        httpCodec.flushRequest();
        realChain.eventListener().responseHeadersStart(realChain.call());
        responseBuilder = httpCodec.readResponseHeaders(true);
  }
  ...
  CountingSink requestBodyOut =
            new CountingSink(httpCodec.createRequestBody(request, contentLength));
  BufferedSink bufferedRequestBody = Okio.buffer(requestBodyOut);

  request.body().writeTo(bufferedRequestBody);

  //刷新socket缓冲区
  httpCodec.finishRequest();

  ...
  //接收响应
  response = response.newBuilder()
          .body(httpCodec.openResponseBody(response))
          .build();
}
  • 请求体的内容写到Okio创建BufferedSink
  • 将缓冲区的数据放入Socket，发出
  • 接收响应BufferSource
  • 更新Socket状态,如果Socket没有新的Stream需要处理，或者没有空连接，关闭当前Socket

连接池的管理

ConnectionPool内部维护着一个双端队列，用于存储连接。还有一个黑名单路由表。还有一个线程池负责回收失效的连接

1
2
private final Deque<RealConnection> connections = new ArrayDeque<>();
final RouteDatabase routeDatabase = new RouteDatabase();

每当有新的连接要放入连接池的时候，会先触发回收连接操作

1
2
3
4
5
6
7
8
9
void put(RealConnection connection) {
    assert (Thread.holdsLock(this));
    if (!cleanupRunning) {
      cleanupRunning = true;
      executor.execute(cleanupRunnable);
    }
    connections.add(connection);
  }

  • 如果清理线程没有再运行，执行清理线程，然后将新的连接添加到连接池中
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
private final Runnable cleanupRunnable = new Runnable() {
    @Override public void run() {
      while (true) {
        long waitNanos = cleanup(System.nanoTime());
        if (waitNanos == -1) return;
        if (waitNanos > 0) {
          long waitMillis = waitNanos / 1000000L;
          waitNanos -= (waitMillis * 1000000L);
          synchronized (ConnectionPool.this) {
            try {
              ConnectionPool.this.wait(waitMillis, (int) waitNanos);
            } catch (InterruptedException ignored) {
            }
          }
        }
      }
    }
  };
  • 线程执行一个死循环，只有当无连接可清理时，线程退出,每次清除，线程会等待一段时间
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
long cleanup(long now) {
  ...
  if (pruneAndGetAllocationCount(connection, now) > 0) {
      inUseConnectionCount++;
      continue;
  }
  ...
  if (idleDurationNs > longestIdleDurationNs) {
    longestIdleDurationNs = idleDurationNs;
    longestIdleConnection = connection;
  }
  ...
  if (longestIdleDurationNs >= this.keepAliveDurationNs
          || idleConnectionCount > this.maxIdleConnections) {
    // We've found a connection to evict. Remove it from the list, then close it below (outside
    // of the synchronized block).
    connections.remove(longestIdleConnection);
  } 
}
  • 寻找有多少连接在使用中，未使用的有多少
  • 找出最长空闲连接，时间必须大于Long.MIN_VALUE
  • 如果最长空闲时间大于最长存活时间或者空闲个数大于最大空闲个数，那么移除当前连接
  • 有空闲连接，即将需要清理，返回需要等待的时间，等下一次清理
  • 所有的连接都在用，返回最长的存活时间，让线程等待
  • 最大空闲连接5个，最长存活时间5分钟

如何知道连接空闲与否?

1
2
3
4
5
private int pruneAndGetAllocationCount(RealConnection connection, long now) {
  ...
  Reference<StreamAllocation> reference = references.get(i);
  ...
}
  • 通过获取连接流配置信息的弱引用。来统计当前的连接是否再用。

流程图

okhttp3

缓存

采用DiskLruCache来缓存请求和响应

写入缓存put
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
private CacheRequest put(Response response) {
    String requestMethod = response.request().method();

    if (HttpMethod.invalidatesCache(response.request().method())) {
      try {
        remove(response.request());
      } catch (IOException ignored) {
        // The cache cannot be written.
      }
      return null;
    }
    if (!requestMethod.equals("GET")) {
      // Don't cache non-GET responses. We're technically allowed to cache
      // HEAD requests and some POST requests, but the complexity of doing
      // so is high and the benefit is low.
      return null;
    }

    if (HttpHeaders.hasVaryAll(response)) {
      return null;
    }

    Entry entry = new Entry(response);
    DiskLruCache.Editor editor = null;
    try {
      editor = cache.edit(urlToKey(response.request()));
      if (editor == null) {
        return null;
      }
      entry.writeTo(editor);
      return new CacheRequestImpl(editor);
    } catch (IOException e) {
      abortQuietly(editor);
      return null;
    }
  }

主要功能：

  1. 检查请求类型，如果不是Get，不缓存
  2. 使用Request的url作为key来缓存
  3. 写入缓存

更新缓存update

1
2
3
4
5
6
7
8
9
10
11
12
13
14
private void update(Response cached, Response network) {
    Entry entry = new Entry(network);
    DiskLruCache.Snapshot snapshot = ((CacheResponseBody) cached.body()).snapshot;
    DiskLruCache.Editor editor = null;
    try {
      editor = snapshot.edit(); // Returns null if snapshot is not current.
      if (editor != null) {
        entry.writeTo(editor);
        editor.commit();
      }
    } catch (IOException e) {
      abortQuietly(editor);
    }
  }

主要工作：

  1. 获取缓存快照编辑器
  2. 将网络响应写入