博客内容Blog Content
一些免费IP解析接口的整合 Integration of free IP Lookup APIs
正在研究一个爬虫程序,能把一些开源免费解析IP接口进行整合,进行统一使用 Working on a program of web crawler to combine several open-source free IP lookup APIs for unified usage
Github Link:https://github.com/luguanxing/FreeIPParsers-Combiner/tree/main
背景 Background
免费就是硬道理! 在互联网上,免费IP解析资源具有不可替代的重要性,但在需要处理大量数据时,选择付费服务可能会带来不小的成本压力。 然而,免费IP解析服务通常存在一些缺点,例如返回的数据字段不完全统一,以及各类使用限制(例如每日调用次数限制等)。
Free is justice! On the internet, free IP parsing resources are of irreplaceable importance, especially when dealing with large volumes of data, where opting for paid services can lead to significant cost pressures. However, free IP parsing services often come with certain drawbacks, such as inconsistencies in the returned data fields and various usage limitations (e.g., daily call limits).
项目介绍 Project Introduction
FreeIPParsers-Combiner 项目的初衷便是解决这些问题。这是一个用 Java 编写的开源项目,通过将多个免费IP 解析接口整合到一起,从而构建一个稳定、能用的免费IP解析器。
The FreeIPParsers-Combiner project was initiated to address these challenges. This is an open-source project written in Java that integrates multiple free IP parsing interfaces to build a functional and stable free IP parser.
项目特点 Project Features
支持对免费IP解析器的灵活扩展,只需要实现”抽取ipData“接口和"解析ipData"两个接口以及返回一个接口的”支持字段列表“即可扩展
实现了线程安全的解析池,内含自动调度机制,对每个解析失败IP的多次重试,以及移除多次连续失败的解析器
Supports flexible extension of free IP parsers. To extend, you only need to implement the "extract ipData" interface and the "parse ipData" interface, as well as return a "supported fields list" interface.
Implemented a thread-safe parsing pool with an automatic scheduling mechanism, including multiple retries for each failed IP parsing and the removal of parsers that fail consecutively multiple times.
Code Example
添加一个解析器只需要实现以下接口;
To add a parser, you only need to implement the following interfaces;
public interface IpParser { Set<IpinfoEnum> getSupportedFields(); JSONObject fetchIpData(String ip); IpInfo parseIpData(JSONObject json); default IpInfo getIpInfo(String ip) { JSONObject ipData = fetchIpData(ip); if (ipData == null) { return null; } return parseIpData(ipData); } }
使用IP解析的示例代码如下:
The example usage code of parsing IPs is as follows:
List<String> ipList = InoutUtil.readIpFile("/data/ips.txt"); List<IpParser> conditionalParsers = ParserFactory.getConditionalParsers(Arrays.asList( IpinfoEnum.IP, IpinfoEnum.COUNTRY, IpinfoEnum.COUNTRY_CODE, IpinfoEnum.REGION, IpinfoEnum.CITY )); List<IpInfo> ipInfos = ExecutorUtil.runParsers(ipList, conditionalParsers, 1); InoutUtil.writeResultFile("/data/ipinfos.txt", ipInfos);