这篇文章总结了我在使用httpclient时候的关键知识点和使用经验。
根据业务量级决定使用同步调用或异步调用:异步回调方式的并发性非常高,缺点是代码可读性一般,在开发中,我会首先选择同步实现,在遇到性能问题后再考虑优化为异步回调方式。在Spring项目中使用HttpClient时,可以借用FactoryBean的概念 ,编写自己的HttpClientFactoryBean,我在LeanJava中写了一个例子:link
同步HttpClient首先编写HttpClientFactoryBean,代码和其中关键的几个参数的解释如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 package org.java.learn.httpclient; import org.apache.commons.codec.Charsets; import org.apache.http.NoHttpResponseException; import org.apache.http.client.ClientProtocolException; import org.apache.http.client.HttpClient; import org.apache.http.client.config.RequestConfig; import org.apache.http.config.ConnectionConfig; import org.apache.http.conn.ConnectTimeoutException; import org.apache.http.impl.client.HttpClients; import org.springframework.beans.factory.FactoryBean; import java.net.SocketTimeoutException; /** * Created by IntelliJ IDEA. * User: duqi * Date: 2017/2/9 * Time: 13:54 */ public class HttpClientFactoryBean implements FactoryBean<HttpClient> { // 知识点1:路由(MAX_PER_ROUTE)是对最大连接数(MAX_TOTAL)的细分,整个连接池的限制数量实际使用DefaultMaxPerRoute并非MaxTotal。 // 设置过小无法支持大并发(ConnectionPoolTimeoutException: Timeout waiting for connection from pool), private static final int DEFAULT_MAX_TOTAL = 512; //最大支持的连接数 private static final int DEFAULT_MAX_PER_ROUTE = 64; //针对某个域名的最大连接数 private static final int DEFAULT_CONNECTION_TIMEOUT = 5000; //知识点2:跟目标服务建立连接超时时间,根据自己的业务调整 private static final int DEFAULT_SOCKET_TIMEOUT = 3000; //知识点3:请求的超时时间(建联后,获取response的返回等待时间) private static final int DEFAULT_TIMEOUT = 1000; //知识点4:从连接池中获取连接的超时时间 @Override public HttpClient getObject() throws Exception { ConnectionConfig config = ConnectionConfig.custom() .setCharset(Charsets.UTF_8) .build(); RequestConfig defaultRequestConfig = RequestConfig.custom() .setConnectTimeout(DEFAULT_CONNECTION_TIMEOUT) .setSocketTimeout(DEFAULT_SOCKET_TIMEOUT) .setConnectionRequestTimeout(DEFAULT_TIMEOUT) .build(); return HttpClients.custom() .setMaxConnPerRoute(DEFAULT_MAX_PER_ROUTE) .setMaxConnTotal(DEFAULT_MAX_TOTAL) .setRetryHandler((exception, executionCount, context) -> executionCount <= 3 && (exception instanceof NoHttpResponseException || exception instanceof ClientProtocolException || exception instanceof SocketTimeoutException || exception instanceof ConnectTimeoutException)) .setDefaultConnectionConfig(config) .setDefaultRequestConfig(defaultRequestConfig) .build(); } @Override public Class<?> getObjectType() { return HttpClient.class; } @Override public boolean isSingleton() { return true; } }
第二,在xml文件中进行如下配置,配置完这一步后,就可以在其他spring bean中编入httpclient使用了。
1 2 3 4 5 6 7 8 9 10 <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd"> <context:component-scan base-package="org.java.learn"/> <bean id="httpClientFactoryBean" class="org.java.learn.httpclient.HttpClientFactoryBean"/> </beans>
第三,编写单元测试,检查是否可用
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 import org.apache.http.client.HttpClient; import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; import javax.annotation.Resource; /** * Created by IntelliJ IDEA. * User: duqi * Date: 2017/2/9 * Time: 14:18 */ @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = "classpath:applicationContext.xml") public class HttpClientFactoryBeanXmlTest { @Resource HttpClient httpClient; @Test public void httpClientAutoWired() throws Exception { Assert.assertNotNull(httpClient); } }
异步HttpClient首先编写AsyncHttpClientFactoryBean,几个关于超时时间的参数和之前相同。这里需要简单理解ioReactor的含义——Async HttpClient使用了Reactor模式,该模式又有别名Dispatcher或Notifier。
从Netty源码解读(四)Netty与Reactor模式 一文可以看到,在Reactor模式中,有一个不断循环的线程监听一个队列,每个异步请求发出去以后,就会在这个队列里注册一个handler(call back对象),当某个请求响应回来后,由中间人负责调用对应的handler,这个中间人的名字就是Reactor。
AsyncHttpClientFactoryBean的代码如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 package org.java.learn.httpclient; import org.apache.commons.lang3.concurrent.BasicThreadFactory; import org.apache.http.client.config.RequestConfig; import org.apache.http.impl.nio.client.CloseableHttpAsyncClient; import org.apache.http.impl.nio.client.HttpAsyncClients; import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager; import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor; import org.apache.http.impl.nio.reactor.IOReactorConfig; import org.springframework.beans.factory.FactoryBean; /** * Created by IntelliJ IDEA. * User: duqi * Date: 2017/2/9 * Time: 15:06 */ public class AsyncHttpClientFactoryBean implements FactoryBean<CloseableHttpAsyncClient> { private static final int DEFAULT_MAX_TOTAL = 512; private static final int DEFAULT_MAX_PER_ROUTE = 64; private static final int DEFAULT_CONNECTION_TIMEOUT = 5000; private static final int DEFAULT_SOCKET_TIMEOUT = 3000; private static final int DEFAULT_TIMEOUT = 1000; @Override public CloseableHttpAsyncClient getObject() throws Exception { DefaultConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(IOReactorConfig.custom() .setSoKeepAlive(true).build()); PoolingNHttpClientConnectionManager pcm = new PoolingNHttpClientConnectionManager(ioReactor); pcm.setMaxTotal(DEFAULT_MAX_TOTAL); pcm.setDefaultMaxPerRoute(DEFAULT_MAX_PER_ROUTE); RequestConfig defaultRequestConfig = RequestConfig.custom() .setConnectTimeout(DEFAULT_CONNECTION_TIMEOUT) .setSocketTimeout(DEFAULT_SOCKET_TIMEOUT) .setConnectionRequestTimeout(DEFAULT_TIMEOUT) .build(); return HttpAsyncClients.custom() .setThreadFactory(new BasicThreadFactory.Builder().namingPattern("AysncHttpThread-%d").build()) .setConnectionManager(pcm) .setDefaultRequestConfig(defaultRequestConfig) .build(); } @Override public Class<?> getObjectType() { return CloseableHttpAsyncClient.class; } @Override public boolean isSingleton() { return true; } }
和之前一样,我们在单元测试中测试了,该FactoryBean已经可以正常为我们生产CloseableHttpAsyncClient对象,现在需要看下如何使用该对象:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 private static final Semaphore concurrency = new Semaphore(1024); @Test public void asyncClientTest() throws Exception { Assert.assertNotNull(asyncClient); //step1 获取信号量控制并发数(防止内存溢出) concurrency.acquireUninterruptibly(); try { //step2 设置HttpUrlRequest final HttpUriRequest httpUriRequest = RequestBuilder.get() .setUri("http://www.baidu.com") .build(); //step3 执行异步调用 asyncClient.execute(httpUriRequest, new FutureCallback<HttpResponse>() { @Override public void completed(HttpResponse httpResponse) { //处理Http响应 } @Override public void failed(Exception e) { //根据情况进行重试 } @Override public void cancelled() { //记录失败日志 } }); } finally { //step4 释放信号量 concurrency.release(); } }
上面四步就是我们使用异步httpclient的常规模式,这里需要使用信号量控制并发,原因是:中间人(Reactor)维护的handler队列是一个无界队列,如果目标服务挂了,这边的请求并发量又很高,就会造成队列无限增长,从而造成OOM。
参考文章使用httpclient必须知道的参数设置及代码写法、存在的风险 ConnectionTimeout, SocketTimeout values set are not effective ](http://stackoverflow.com/questions/9725492/java-httpclient-4-1-2-connectiontimeout-sockettimeout-values-set-are-not-ef )Thoughts, stories and ideas.