Android Netroid解析之——断点续传下载及问题修正

提到Netroid或许很多人不知道这个框架，但我如果说Volley想必没有人不知道吧。Netroid是一个基于Volley实现的Android Http库。提供执行网络请求、缓存返回结果、批量图片加载、大文件断点下载的常见Http交互功能，关于网络请求，图片加载没什么好说的，Volley已经有很多人解析过了，这里来说一下大文件断点下载。

关于大文件断点下载，网上也有很多实现的demo,为什么要单单说Netroid呢？因为Netroid断点续传不依赖数据库，我在网上看到过很多的断点续传的例子，无一例外都是依赖于数据库，包括DownloadManager，大名鼎鼎的xutils,但是这两个都有一定的问题。

1.DownloadManager在三星手机上必须打开下载管理才能应用，而打开这个管理必须需要手动打开，一般情况下无伤大雅，视情况而定

2.xutils这个框架别的不知道，文件下载这块慎用

好了，进入正题，Netroid的地址:https://github.com/vince-styling/,下面简单的说一下这个框架文件下载的实现和原理,

	// 1
		RequestQueue queue = Netroid.newRequestQueue(getApplicationContext(), null);
		// 2
		mDownloder = new FileDownloader(queue, 1) {
			@Override
			public FileDownloadRequest buildRequest(String storeFilePath, String url) {
				return new FileDownloadRequest(storeFilePath, url) {
					@Override
					public void prepare() {
						addHeader("Accept-Encoding", "identity");
						super.prepare();
					}
				};
			}
		};
		// 3
		task.controller = mDownloder.add(mSaveDirPath + task.storeFileName, task.url, new Listener<Void>() {
			@Override
			public void onPreExecute() {
				task.invalidate();
			}

			@Override
			public void onSuccess(Void response) {
				showToast(task.storeFileName + " Success!");
			}

			@Override
			public void onError(NetroidError error) {
				NetroidLog.e(error.getMessage());
			}

			@Override
			public void onFinish() {
				NetroidLog.e("onFinish size : " + Formatter.formatFileSize(
						FileDownloadActivity.this, new File(mSaveDirPath + task.storeFileName).length()));
				task.invalidate();
			}

			@Override
			public void onProgressChange(long fileSize, long downloadedSize) {
				task.onProgressChange(fileSize, downloadedSize);
//				NetroidLog.e("---- fileSize : " + fileSize + " downloadedSize : " + downloadedSize);
			}
		});

1.创建一个请求队列

2.构建一个文件下载管理器

3.将下载任务添加到队列

现在根据上面的三步来看一下它的实现原理：

第一步：创建一个请求队列:RequestQueue queue = Netroid.newRequestQueue(getApplicationContext(), null);

/**
     * Creates a default instance of the worker pool and calls {@link RequestQueue#start()} on it.
     * @param context A {@link Context} to use for creating the cache dir.
     * @return A started {@link RequestQueue} instance.
     */
    public static RequestQueue newRequestQueue(Context context, DiskCache cache) {
		int poolSize = RequestQueue.DEFAULT_NETWORK_THREAD_POOL_SIZE;

		HttpStack stack;
		String userAgent = "netroid/0";
		try {
			String packageName = context.getPackageName();
			PackageInfo info = context.getPackageManager().getPackageInfo(packageName, 0);
			userAgent = packageName + "/" + info.versionCode;
		} catch (NameNotFoundException e) {
		}

		if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.GINGERBREAD) {
			stack = new HurlStack(userAgent, null);
		} else {
			// Prior to Gingerbread, HttpUrlConnection was unreliable.
			// See: http://android-developers.blogspot.com/2011/09/androids-http-clients.html
			stack = new HttpClientStack(userAgent);
		}
		//实例化BasicNetwork，主要用于执行下载请求
		Network network = new BasicNetwork(stack, HTTP.UTF_8);
		//创建请求队列
		RequestQueue queue = new RequestQueue(network, poolSize, cache);
		//很重要的一步
		queue.start();

        return queue;
    }

  /**
     * Starts the dispatchers in this queue.
     */
    public void start() {
        stop();  // Make sure any currently running dispatchers are stopped.
        // Create the cache dispatcher and start it.
        mCacheDispatcher = new CacheDispatcher(mCacheQueue, mNetworkQueue, mCache, mDelivery);
        mCacheDispatcher.start();

        // Create network dispatchers (and corresponding threads) up to the pool size.
        for (int i = 0; i < mDispatchers.length; i++) {
        	//一个线程，从请求队列中获取任务并执行
            NetworkDispatcher networkDispatcher =
					new NetworkDispatcher(mNetworkQueue, mNetwork, mCache, mDelivery);
            mDispatchers[i] = networkDispatcher;
            //Thread run()
            networkDispatcher.start();
        }
    }

    /**
     * Stops the cache and network dispatchers.
     */
    public void stop() {
        if (mCacheDispatcher != null) {
            mCacheDispatcher.quit();
        }
		for (NetworkDispatcher mDispatcher : mDispatchers) {
			//Thread interrupt()线程中断
			if (mDispatcher != null) mDispatcher.quit();
		}
    }

框架中对于文件是没有缓存机制的，所以mCacheDispatcher可以不用理它，看一下NetworkDispatcher这个线程做了什么：com.duowan.mobile.netroid.NetworkDispatcher

public class NetworkDispatcher extends Thread {

    @Override
    public void run() {
    	//设置线程优先级
        Process.setThreadPriority(Process.THREAD_PRIORITY_BACKGROUND);
        Request request;
        while (true) {
            try {
                // Take a request from the queue.如果队列为空，则阻塞
                request = mQueue.take();
            } catch (InterruptedException e) {
                // We may have been interrupted because it was time to quit.唯有线程中断的时候mQuit才为true,InterruptedException为中断异常
            	//mQueue.take()如果队列为null,只会阻塞，不会跑出异常
                if (mQuit) return;
                continue;
            }

            try {
                request.addMarker("network-queue-take");
                //准备执行
				mDelivery.postPreExecute(request);

                // If the request was cancelled already,
                // do not perform the network request.
                if (request.isCanceled()) {
                    request.finish("network-discard-cancelled");
					mDelivery.postCancel(request);
					mDelivery.postFinish(request);
                    continue;
                }

                // Perform the network request.最重要一步！Netroid实例化的BasicNetwork在这里执行网络请求
                NetworkResponse networkResponse = mNetwork.performRequest(request);
                request.addMarker("network-http-complete");

                // Parse the response here on the worker thread.重命名一下，没做什么
                Response<?> response = request.parseNetworkResponse(networkResponse);
                request.addMarker("network-parse-complete");

                // Write to cache if applicable.
				if (mCache != null && request.shouldCache() && response.cacheEntry != null) {
					response.cacheEntry.expireTime = request.getCacheExpireTime();
					mCache.putEntry(request.getCacheKey(), response.cacheEntry);
					request.addMarker("network-cache-written");
				}

                // Post the response back.
                request.markDelivered();
                mDelivery.postResponse(request, response);
            } catch (NetroidError netroidError) {
				mDelivery.postError(request, request.parseNetworkError(netroidError));
            } catch (Exception e) {
				NetroidLog.e(e, "Unhandled exception %s", e.toString());
				mDelivery.postError(request, new NetroidError(e));
			}
        }
    }

}

这里最重要的一步就是NetworkResponse networkResponse = mNetwork.performRequest(request);执行网络请求，但是我们不要忘记我们的mQueue还是空的，mQueue.take()正在阻塞着呢，所以，现在还没有办法进行网络请求，因此我们需要在mQueue中填充任务，才能进行我们的网络请求。不要忘记这里哦，因为我们还会回到这里！

第二步：创建一个文件下载管理器：new FileDownloader(queue, 1)

mDownloder = new FileDownloader(queue, 1) {
			@Override
			public FileDownloadRequest buildRequest(String storeFilePath, String url) {
				return new FileDownloadRequest(storeFilePath, url) {
					@Override
					public void prepare() {
						addHeader("Accept-Encoding", "identity");
						super.prepare();
					}
				};
			}
		};

	/** The parallel task count, recommend less than 3. */
	private final int mParallelTaskCount;

	/** The linked Task Queue. */
	private final LinkedList<DownloadController> mTaskQueue;

	/**
	 * Construct Downloader and init the Task Queue.
	 * @param queue The RequestQueue for dispatching Download task.
	 * @param parallelTaskCount
	 * 				Allows parallel task count,
	 * 				don't forget the value must less than ThreadPoolSize of the RequestQueue.
	 */
	public FileDownloader(RequestQueue queue, int parallelTaskCount) {
		if (parallelTaskCount >= queue.getThreadPoolSize()) {
			throw new IllegalArgumentException("parallelTaskCount[" + parallelTaskCount
					+ "] must less than threadPoolSize[" + queue.getThreadPoolSize() + "] of the RequestQueue.");
		}

		mTaskQueue = new LinkedList<DownloadController>();
		mParallelTaskCount = parallelTaskCount;
		mRequestQueue = queue;
	}

第三步：将下载任务添加到队列，task.controller = mDownloder.add(mSaveDirPath + task.storeFileName, task.url, new Listener<Void>()：

	/**
	 * Create a new download request, this request might not run immediately because the parallel task limitation,
	 * you can check the status by the {@link DownloadController} which you got after invoke this method.
	 *
	 * Note: don't perform this method twice or more with same parameters, because we didn't check for
	 * duplicate tasks, it rely on developer done.
	 *
	 * Note: this method should invoke in the main thread.
	 *
	 * @param storeFilePath Once download successed, we'll find it by the store file path.
	 * @param url The download url.
	 * @param listener The event callback by status;
	 * @return The task controller allows pause or resume or discard operation.
	 */
	public DownloadController add(String storeFilePath, String url, Listener<Void> listener) {
		// only fulfill requests that were initiated from the main thread.(reason for the Delivery?)
		//看名字就知道
		throwIfNotOnMainThread();
		//创建一个下载控制器
		DownloadController controller = new DownloadController(storeFilePath, url, listener);
		synchronized (mTaskQueue) {
			//这可不是mQueue,这里只是一个DownloadController的LinkedList集合
			mTaskQueue.add(controller);
		}
		//重点来了
		schedule();
		return controller;
	}

	/**
	 * Traverse the Task Queue, count the running task then deploy more if it can be.
	 */
	private void schedule() {
		// make sure only one thread can manipulate the Task Queue.
		synchronized (mTaskQueue) {
			// counting ran task.
			int parallelTaskCount = 0;
			for (DownloadController controller : mTaskQueue) {
				//累计队列中正在下载的的任务数
				if (controller.isDownloading()) parallelTaskCount++;
			}
			//当正在下载的个数大于并行任务数的时候，不在执行下载任务
			/*
			 * 这里举个例子说明一下：我们默认mParallelTaskCount=1
			 * 当我们添加第一个任务的时候，这个的controller.isDownloading()肯定是false
			 * 所以parallelTaskCount >= mParallelTaskCount是不成立的，当我们再添加一个任务的时候，现在mTaskQueue.size是2了
			 * 且第一个isDownloading，为了保证并发数量为1，会return,说的有点乱，不知道说明白了没有
			 */
			if (parallelTaskCount >= mParallelTaskCount) return;

			// try to deploy all Task if they're await.
			for (DownloadController controller : mTaskQueue) {
				//deploy()，将任务添加到队列中
				if (controller.deploy() && ++parallelTaskCount == mParallelTaskCount) return;
			}
		}
	}

		/**
		 * For the parallel reason, only the {@link FileDownloader#schedule()} can call this method.
		 * @return true if deploy is successed.
		 */
		private boolean deploy() {
			if (mStatus != STATUS_WAITING) return false;
			//第二步我说很吓人那个地方
			mRequest = buildRequest(mStoreFilePath, mUrl);

			// we create a Listener to wrapping that Listener which developer specified,
			// for the onFinish(), onSuccess(), onError() won't call when request was cancel reason.
			mRequest.setListener(new Listener<Void>() {
				boolean isCanceled;

				@Override
				public void onPreExecute() {
					mListener.onPreExecute();
				}

				@Override
				public void onFinish() {
					// we don't inform FINISH when it was cancel.
					if (!isCanceled) {
						mStatus = STATUS_PAUSE;
						mListener.onFinish();
						// when request was FINISH, remove the task and re-schedule Task Queue.
//						remove(DownloadController.this);
					}
				}

				@Override
				public void onSuccess(Void response) {
					// we don't inform SUCCESS when it was cancel.
					if (!isCanceled) {
						mListener.onSuccess(response);
						mStatus = STATUS_SUCCESS;
						remove(DownloadController.this);
					}
				}

				@Override
				public void onError(NetroidError error) {
					// we don't inform ERROR when it was cancel.
					if (!isCanceled) mListener.onError(error);
				}

				@Override
				public void onCancel() {
					mListener.onCancel();
					isCanceled = true;
				}

				@Override
				public void onProgressChange(long fileSize, long downloadedSize) {
					mListener.onProgressChange(fileSize, downloadedSize);
				}
			});

			mStatus = STATUS_DOWNLOADING;
			//我擦，终于把任务加到队列中了
			mRequestQueue.add(mRequest);
			return true;
		}

 public Request add(Request request) {
        // Tag the request as belonging to this queue and add it to the set of current requests.
        request.setRequestQueue(this);
        synchronized (mCurrentRequests) {
            mCurrentRequests.add(request);
        }

        // Process requests in the order they are added.
        request.setSequence(getSequenceNumber());
        request.addMarker("add-to-queue");

        // If the request is uncacheable or forceUpdate, skip the cache queue and go straight to the network.
        if (request.isForceUpdate() || !request.shouldCache()) {
			mDelivery.postNetworking(request);
			mNetworkQueue.add(request);
			return request;
        }

}

request.shouldCache()有兴趣的可以自己去看一下，这里说明了文件下载没有缓存机制，这里就不多说了，因为如果你还没有忘记的话，mQueue.take()还在阻塞着呢，好了让我们回到第一步，执行网络请求

 NetworkResponse networkResponse = mNetwork.performRequest(request);

	@Override
	public NetworkResponse performRequest(Request<?> request) throws NetroidError {
		// Determine if request had non-http perform.
		NetworkResponse networkResponse = request.perform();
		if (networkResponse != null) return networkResponse;

		long requestStart = SystemClock.elapsedRealtime();
		while (true) {
			// If the request was cancelled already,
			// do not perform the network request.
			if (request.isCanceled()) {
				request.finish("perform-discard-cancelled");
				mDelivery.postCancel(request);
				throw new NetworkError(networkResponse);
			}

			HttpResponse httpResponse = null;
			byte[] responseContents = null;
			try {
				// prepare to perform this request, normally is reset the request headers.
				request.prepare();

				httpResponse = mHttpStack.performRequest(request);

				StatusLine statusLine = httpResponse.getStatusLine();
				int statusCode = statusLine.getStatusCode();
				responseContents = request.handleResponse(httpResponse, mDelivery);
				if (statusCode < 200 || statusCode > 299) throw new IOException();


				// if the request is slow, log it.
				long requestLifetime = SystemClock.elapsedRealtime() - requestStart;
				logSlowRequests(requestLifetime, request, responseContents, statusLine);

				return new NetworkResponse(statusCode, responseContents, parseCharset(httpResponse));
			} catch (SocketTimeoutException e) {
				attemptRetryOnException("socket", request, new TimeoutError());
			} catch (ConnectTimeoutException e) {
				attemptRetryOnException("connection", request, new TimeoutError());
			} catch (MalformedURLException e) {
				throw new RuntimeException("Bad URL " + request.getUrl(), e);
			} catch (IOException e) {
				if (httpResponse == null) throw new NoConnectionError(e);

				int statusCode = httpResponse.getStatusLine().getStatusCode();
				NetroidLog.e("Unexpected response code %d for %s", statusCode, request.getUrl());
				if (responseContents != null) {
					networkResponse = new NetworkResponse(statusCode, responseContents, parseCharset(httpResponse));
					if (statusCode == HttpStatus.SC_UNAUTHORIZED || statusCode == HttpStatus.SC_FORBIDDEN) {
						attemptRetryOnException("auth", request, new AuthFailureError(networkResponse));
					} else {
						// TODO: Only throw ServerError for 5xx status codes.
						throw new ServerError(networkResponse);
					}
				} else {
					throw new NetworkError(networkResponse);
				}
			}
		}
	}

responseContents = request.handleResponse(httpResponse, mDelivery);，写文件,断点续传的原理

	/**
	 * In this method, we got the Content-Length, with the TemporaryFile length,
	 * we can calculate the actually size of the whole file, if TemporaryFile not exists,
	 * we'll take the store file length then compare to actually size, and if equals,
	 * we consider this download was already done.
	 * We used {@link RandomAccessFile} to continue download, when download success,
	 * the TemporaryFile will be rename to StoreFile.
	 */
	@Override
	public byte[] handleResponse(HttpResponse response, Delivery delivery) throws IOException, ServerError {
		// Content-Length might be negative when use HttpURLConnection because it default header Accept-Encoding is gzip,
		// we can force set the Accept-Encoding as identity in prepare() method to slove this problem but also disable gzip response.
		HttpEntity entity = response.getEntity();
		//获取文件的总大小
		long fileSize = entity.getContentLength();
		if (fileSize <= 0) {
			NetroidLog.d("Response doesn't present Content-Length!");
		}
		
		long downloadedSize = mTemporaryFile.length();
		/*
		 * 是否支持断点续传
		 * 
		 * 客户端每次提交下载请求时，服务端都要添加这两个响应头，以保证客户端和服务端将此下载识别为可以断点续传的下载：
		 *  Accept-Ranges：告知下载客户端这是一个可以恢复续传的下载，存放本次下载的开始字节位置、文件的字节大小；
		 *  ETag：保存文件的唯一标识（我在用的文件名+文件最后修改时间，以便续传请求时对文件进行验证）；
		 *  Last-Modified：可选响应头，存放服务端文件的最后修改时间，用于验证
		 */
		boolean isSupportRange = HttpUtils.isSupportRange(response);
		if (isSupportRange) {
			fileSize += downloadedSize;

			// Verify the Content-Range Header, to ensure temporary file is part of the whole file.
			// Sometime, temporary file length add response content-length might greater than actual file length,
			// in this situation, we consider the temporary file is invalid, then throw an exception.
			String realRangeValue = HttpUtils.getHeader(response, "Content-Range");
			// response Content-Range may be null when "Range=bytes=0-"
			if (!TextUtils.isEmpty(realRangeValue)) {
				String assumeRangeValue = "bytes " + downloadedSize + "-" + (fileSize - 1);
				if (TextUtils.indexOf(realRangeValue, assumeRangeValue) == -1) {
					throw new IllegalStateException(
							"The Content-Range Header is invalid Assume[" + assumeRangeValue + "] vs Real[" + realRangeValue + "], " +
									"please remove the temporary file [" + mTemporaryFile + "].");
				}
			}
		}

		// Compare the store file size(after download successes have) to server-side Content-Length.
		// temporary file will rename to store file after download success, so we compare the
		// Content-Length to ensure this request already download or not.
		if (fileSize > 0 && mStoreFile.length() == fileSize) {
			// Rename the store file to temporary file, mock the download success. ^_^
			mStoreFile.renameTo(mTemporaryFile);

			// Deliver download progress.
			delivery.postDownloadProgress(this, fileSize, fileSize);

			return null;
		}
		//之所以能够实现断点续传的原因所在
		RandomAccessFile tmpFileRaf = new RandomAccessFile(mTemporaryFile, "rw");

		// If server-side support range download, we seek to last point of the temporary file.
		if (isSupportRange) {
			//移动文件读写指针位置
			tmpFileRaf.seek(downloadedSize);
		} else {
			// If not, truncate the temporary file then start download from beginning.
			tmpFileRaf.setLength(0);
			downloadedSize = 0;
		}

		try {
			InputStream in = entity.getContent();
			// Determine the response gzip encoding, support for HttpClientStack download.
			if (HttpUtils.isGzipContent(response) && !(in instanceof GZIPInputStream)) {
				in = new GZIPInputStream(in);
			}
			byte[] buffer = new byte[6 * 1024]; // 6K buffer
			int offset;

			while ((offset = in.read(buffer)) != -1) {
				//写文件
				tmpFileRaf.write(buffer, 0, offset);

				downloadedSize += offset;
				long currTime = SystemClock.uptimeMillis();
				//控制下载进度的速度
				if (currTime - lastUpdateTime >= DEFAULT_TIME) {
					lastUpdateTime = currTime;
					delivery.postDownloadProgress(this, fileSize,
							downloadedSize);
				}

				if (isCanceled()) {
					delivery.postCancel(this);
					break;
				}
			}
		} finally {
			try {
				// Close the InputStream and release the resources by "consuming the content".
				if (entity != null) entity.consumeContent();
			} catch (Exception e) {
				// This can happen if there was an exception above that left the entity in
				// an invalid state.
				NetroidLog.v("Error occured when calling consumingContent");
			}
			tmpFileRaf.close();
		}

		return null;
	}

好了，Netroid的原理基本上就是这些了，讲一下我用的时候遇到的两个问题：

1.下载进度的速度太快，你如果用notifition来显示，会出现ANR,所以我们要控制一下它的速度，具体方法在上面

//控制下载进度的速度
				if (currTime - lastUpdateTime >= DEFAULT_TIME) {
					lastUpdateTime = currTime;
					delivery.postDownloadProgress(this, fileSize,
							downloadedSize);
				}

	@Override
				public void onFinish() {
					// we don't inform FINISH when it was cancel.
					if (!isCanceled) {
						mStatus = STATUS_PAUSE;
						mListener.onFinish();
						// when request was FINISH, remove the task and re-schedule Task Queue.
//						remove(DownloadController.this);
					}
				}

				@Override
				public void onSuccess(Void response) {
					// we don't inform SUCCESS when it was cancel.
					if (!isCanceled) {
						mListener.onSuccess(response);
						mStatus = STATUS_SUCCESS;
						remove(DownloadController.this);
					}
				}

把onFinish的status改成STATUS_PAUSE，并去掉remove(DownloadController.this);，在onSuccess中再将status修改为STATUS_SUCCESS，并remove，当然这个办法治标不治本，如果有谁知道请告之，谢谢！