EP卫星数据预处理与分发服务平台设计与实现
doi: 10.11728/cjss2025.04.2024-0092 cstr: 32142.14.cjss.2024-0092
-
马福利 男, 1986年10月出生, 中国科学院国家空间科学中心高级工程师, 主要从事卫星地面数据处理技术和大数据计算与存储技术的研究等. E-mail: mafuli007@nssc.ac.cn
作者简介:
Design of the EP Satellite Data Preprocessing and Distribution Service Platform
-
摘要: 随着空间探测技术发展和星地通信手段的增加, 为了能够第一时间对重要天文事件进行分析, 空间天文卫星对下行探测数据的实时性和处理效率的要求日益严格. 本文针对爱因斯坦探针(EP)卫星的数传X-band、遥测S-band、甚高频(VHF)数据和北斗短报文多种类型数据高时效性处理与分发需求, 开展了EP卫星数据预处理与分析平台关键技术研究, 设计了载荷0级数据处理、1级数据处理以及基于观测组织的数据完整性判读等关键算法; 针对VHF数据和北斗短报文中天文警报信息的高时效性发布需求, 设计了不同优先级的数据预处理和分发流程, 最终采用分层架构设计的方法研制了EP卫星数据预处理与分发服务平台, 具备对EP卫星下行的各类数据进行自动、高效和准确预处理、管理以及分发的能力. 系统运行验证结果表明, 该平台处理的各级各类科学数据具有较高的准确性, 能够满足EP卫星数据预处理的时效性要求, 且具有良好的可扩展性.Abstract: Along with the development of space exploration technology and the communication methods between satellite and the Earth, the requirements for the efficiency of data transmission and processing in astronomical missions are increasing day by day. The article introduces the design and implementation of a data preprocessing and distribution platform, which focuses on the processing and distribution requirements of various types of data in the Einstein Probe (EP) mission. The EP mission downlinks data via four different channels, including X-band, S-band, VHF channel, and Beidou channel. For each data channel, the data processing methods and efficiency requirements are different, such as the VHF and Beidou data needs to be processed in real-time, the X-band data needs to be processed comprehensively by different types of data packets. During the development process, the key technologies of the EP satellite data preprocessing and analysis platform have been studied. At the meanwhile, several key algorithms have been designed, such as the level 0 data processing method, level 1 data processing method and data product integrity interpretation based on observation organization. Especially, regarding the high timeliness requirements for the release of astronomical alert information in VHF data and Beidou short messages, different priority data preprocessing and distribution workflows were designed. The EP mission data preprocessing and distribution service platform adopts a hierarchical architecture design. The application results indicate that the platform has the ability to preprocess, manage and distribute various types of data in EP mission automatically, efficiently, and accurately. The operational process of the platform supports the different processing flows for different data products and simultaneous data distribution and acquisition for multiple users. The platform has good scalability and efficiency. The various levels and types of scientific data products processed by the platform have high accuracy, can satisfy the application requirements of EP scientific teams.
-
Key words:
- Beidou short message /
- Space science satellite /
- Data preprocessing /
- Data distribution
-
图 3 EP数据处理与分发平台分层架构
Figure 3. EP data processing and distribution platform architecture
图 4 EP卫星数据预处理与分发运行流程
Figure 4. EP data processing and distribution operation flow chart
图 10 EP卫星数据预处理与分发服务平台组成
Figure 10. EP data processing and distribution platform structure
表 1 EP卫星0级和1级数据定义
Table 1. Definition of EP Level 0 and Level 1 data product
数据级别 数据级别定义 Level 0A 对卫星单次过站的数传X-band原始数据、遥测S-band原始数据、北斗短报文以及VHF数据进行解帧、源包提取、验证排序后生成的按应用过程标识符(Application Process Identifier, APID)拆分的源包数据 Level 0B 在0A级数据的基础上, 融合多次数传接收数据, 对数据源包进行拼接、去重复后形成按小时组织, 按APID拆分的数据 Level 0C 在0A/0B级科学数据的基础上, 进行拼接、去重复、源包解析后生成按观测号组织的科学数据帧数据 Level 0Q 在0B/0C级科学数据的基础上, 对载荷科学数据进行粗略标定后生成的光变曲线、曝光图以及能谱数据 Level 1 FXT载荷: 在0C级科学数据的基础上, 进行科学数据解包、时间还原和坐标系转换, 对同一观测号的数据进行再组织, 剔选出和科学分析无关数据
WXT载荷: 在0B级科学数据的基础上, 进行解包、科学数据帧解析、物理量转化, 按观测号组织, 同时整合载荷状态数据、姿轨信息等得到的数据
下载: 导出CSV
表 2 EP卫星各类数据处理与分发时效性
Table 2. EP data processing and distribution performance
数据来源 数据类型 数据级别 数据体量 数据个数 处理延迟 分发延迟 数传数据 科学 0B 1484.8 MByte 5 45 min 23 min 1级 3686.4 MByte 19 69 min 1.6 min 工程 0B 146 MByte 185 45 min 24 min 1级 902 MByte 193 67 min 18 min 警报 1 A 4.74 MByte 11 14 min 11 min VHF数据 警报 0A 9 KByte 1 21 s 1 s 工程 1级 864 Byte 1 7 s 1 s 北斗数据 警报 0C 8.43 KByte 1 1 s 1 s 工程 1级 1 KByte 1 12 s 1 s 遥测数据 工程 1级 2.44 MByte 36 17 s 4 s
下载: 导出CSV
表 3 EP卫星数据管理详情
Table 3. EP satellite data management
序号 数据级别 种类数量 文件个数 体量 1 原始 4 42666 4.4 TByte 2 0A级 87 273044 1.99 TByte 3 0B级 39 480540 1.78 TByte 4 0C级 7 15125 1.24 TByte 5 1级 94 824349 9.22 TByte 6 辅助 4 1427 102.16 GByte
下载: 导出CSV
表 4 EP卫星观测数据完整性判读结果
Table 4. Observation data integrity interpretation results
序号 观测号 载荷 观测类型 观测开始
时间(UTC)观测结束
时间(UTC)更新时间
(UTC)观测是否
结束工程数据
完整性科学数据
完整性1 08500000099 FXTA 重要ToO 2024-05-18 (15:00:01) 2024-05-18 (18:33:44) 2024-05-19 (10:03:07) 结束 完整 完整 2 13600006196 FXTB 巡天观测 2024-05-17 (21:44:03) 2024-05-19 (18:38:47) 2024-05-20 (05:48:44) 结束 完整 完整 3 13600006197 FXTB 巡天观测 2024-05-17 (20:07:44) 2024-05-17 (21:44:03) 2024-05-18 (14:06:07) 结束 完整 完整 4 08500000098 FXTA 重要ToO 2024-05-17 (13:35:01) 2024-05-17 (20:07:44) 2024-05-19 (09:46:00) 结束 完整 完整 5 13600006196 FXTB 巡天观测 2024-05-16 (10:24:59) 2024-05-17 (13:35:01) 2024-05-18 (10:51:00) 结束 完整 完整 6 08500000097 FXTA 重要ToO 2024-05-16 (02:23:18) 2024-05-16 (10:24:59) 2024-05-18 (08:57:13) 结束 完整 完整
下载: 导出CSV
-
[1] National Research Council. New Worlds, New Horizons in Astronomy and Astrophysics[M]. Washington: The National Academies Press, 2010 [2] ABBOTT B P, ABBOTT R, ABBOTT T D, et al. GW151226: observation of gravitational waves from a 22-solar-mass binary black hole coalescence[J]. Physical Review Letters, 2016, 116(24): 241103 doi: 10.1103/PhysRevLett.116.241103 [3] 袁为民, 张臣, 陈勇, 等. 爱因斯坦探针: 探索变幻多姿的X射线宇宙[J]. 中国科学: 物理学 力学 天文学, 2018, 48(3): 039502YUAN Weimin, ZHANG Chen, CHEN Yong, et al. Einstein probe: exploring the ever-changing X-ray universe[J]. Scientia Sinica Physica, Mechanica & Astronomica, 2018, 48(3): 039502 [4] ZHANG Z, BARBARY K, NOTHAFT F A, et al. Kira: processing astronomy imagery using big data technology[J]. IEEE Transactions on Big Data, 2020, 6(2): 369-381 doi: 10.1109/TBDATA.2016.2599926 [5] 高立京, 陈志敏, 王春梅, 等. 基于Flink的空间科学卫星数据实时处理方法[J]. 计算机仿真, 2023, 40(7): 26-31,290 doi: 10.3969/j.issn.1006-9348.2023.07.005GAO Lijing, CHEN Zhimin, WANG Chunmei, et al. Real-time processing method of space science satellite downlink data based on streaming computing[J]. Computer Simulation, 2023, 40(7): 26-31,290 doi: 10.3969/j.issn.1006-9348.2023.07.005 [6] 王静, 王春梅, 智佳, 等. 面向有效载荷高速数据流的数据处理方法[J]. 计算机工程与设计, 2017, 38(4): 941-945WANG Jing, WANG Chunmei, ZHI Jia, et al. Technology of data processing for high speed data stream of payload[J]. Computer Engineering and Design, 2017, 38(4): 941-945 [7] 张文彬, 王春梅, 王静, 等. 基于Spark的有效载荷参数解析处理方法[J]. 计算机工程与设计, 2020, 41(2): 587-591ZHANG Wenbin, WANG Chunmei, WANG Jing, et al. Processing method of parameter analysis of payload based on spark[J]. Computer Engineering and Design, 2020, 41(2): 587-591 [8] 戚红雨. 流式处理框架发展综述[J]. 信息化研究, 2019, 45(6): 1-8QI Hongyu. Survey of distributed data flow processing framework[J]. Informatization Research, 2019, 45(6): 1-8 [9] KARAKAYA Z, YAZICI A, ALAYYOUB M. A comparison of stream processing frameworks[C]//Proceedings of 2017 International Conference on Computer and Applications (ICCA). Doha, Qatar, 2017: 1-12 [10] ALMEIDA A, BRÁS S, SARGENTO S, et al. Time series big data: a survey on data stream frameworks, analysis and algorithms[J]. Journal of Big Data, 2023, 10(1): 83 doi: 10.1186/s40537-023-00760-1 [11] ALKHANAK E N, LEE S P. A hyper-heuristic cost optimisation approach for Scientific Workflow Scheduling in cloud computing[J]. Future Generation Computer Systems, 2018, 86: 480-506 doi: 10.1016/j.future.2018.03.055 [12] 冯阳, 张国强. 基于工作流技术的遥感卫星数据接收调度系统的设计与实现[J]. 无线电工程, 2018, 48(11): 1004-1008 doi: 10.3969/j.issn.1003-3106.2018.11.18FENG Yang, ZHANG Guoqiang. Design and implementation of earth observation satellite data receiving scheduling system based on workflow technology[J]. Radio Engineering, 2018, 48(11): 1004-1008 doi: 10.3969/j.issn.1003-3106.2018.11.18 [13] 马福利, 陈玲, 李冰, 等. GECAM卫星快速预处理流程设计与实现[J]. 天文研究与技术, 2022, 19(3): 274-282MA Fuli, CHEN Ling, LI Bing, et al. Design and implementation of GECAM preprocessing pipeline[J]. Astronomical Research :Times New Roman;">& Technology, 2022, 19(3): 274-282 [14] 陈亚杰, 王锋, 邓辉, 等. ElasticSearch分布式搜索引擎在天文大数据检索中的应用研究[J]. 天文学报, 2016, 57(2): 241-251CHEN Yajie, WANG Feng, DEGN Hui, et al. The application of elastic search in the massive astronomical data retrieval[J]. Acta Astronomica Sinica, 2016, 57(2): 241-251 -
-
下载:
图(10) / 表(4)
计量
- 文章访问数: 239
- HTML全文浏览量: 18
- PDF下载量: 17
-
被引次数:
0(来源:Crossref)
0(来源:其他)
