#cofacts
2026-01-04
@robert79kimo has joined the channel
問一下cofast工作會議是可以直接去還是要報名
問一下cofast工作會議是可以直接去還是要報名
可以直接來唷
可以直接來唷
2026-01-05
好👌
好👌
@dog850402 has joined the channel
2026-01-06
HackMD
# Cofacts 會議記錄 - [搜尋](<https://cse.google.com/cse?cx=71f4f7ee215d54fe6>)[target=_blank] ## 2026 -
話說怎麼上去
話說怎麼上去
:melting_face:
:melting_face:
感謝
感謝
emmm
emmm
hello
2026-01-08
看到mygopen這篇
https://www.mygopen.com/2026/01/taichung.html
裡面提到有359家診所可以使用敬老愛心卡服務，但是我去翻衛生局網站看有補助50元的名單只有91家
https://www.health.taichung.gov.tw/3181762/post
所以可以用敬老愛心卡的診所不等於有補助？
https://www.mygopen.com/2026/01/taichung.html
裡面提到有359家診所可以使用敬老愛心卡服務，但是我去翻衛生局網站看有補助50元的名單只有91家
https://www.health.taichung.gov.tw/3181762/post
所以可以用敬老愛心卡的診所不等於有補助？
感謝 @iacmai 回報與 Charles 超迅速回應
偉哉 message bridge
看到mygopen這篇
https://www.mygopen.com/2026/01/taichung.html
裡面提到有359家診所可以使用敬老愛心卡服務，但是我去翻衛生局網站看有補助50元的名單只有91家
https://www.health.taichung.gov.tw/3181762/post
所以可以用敬老愛心卡的診所不等於有補助？
https://www.mygopen.com/2026/01/taichung.html
裡面提到有359家診所可以使用敬老愛心卡服務，但是我去翻衛生局網站看有補助50元的名單只有91家
https://www.health.taichung.gov.tw/3181762/post
所以可以用敬老愛心卡的診所不等於有補助？
MyGoPen
網傳「台中敬老卡優惠」的訊息，內容聲稱台中市使用敬老愛心卡的族群，可以在 19 家醫院看診，並且折抵掛號費，並列出適用醫院的清單。經查證，網傳訊息曾是台中市政府於 2018 年「預計」推動的計畫，但從未正式實施；根據現行政策，敬老愛心卡實際適用地點為「衛生所與基層診所」，而非「地區醫院、區域醫院或醫
感謝 @iacmai 回報與 Charles 超迅速回應
偉哉 message bridge
2026-01-09
感謝 @iacmai 回報與 Charles 超迅速回應
偉哉 message bridge
15:21 收到 Cloudflare alert, 15:25 解除狀況
工程方向仍是把 url-resolver 搬出去
這篇沒被 message bridge forward 出去？
cc/ @pm5 bridge 好像怪怪的
好，我下班回家後看一下
好像有點麻煩。可能要明天再來處理
辛苦了 QQ
重開以後在 # 測試恢復了。@mrorz 再試試看吧
15:21 收到 Cloudflare healthcheck alert, 15:25 解除狀況
工程方向仍是把 url-resolver 搬出去
這篇沒被 message bridge forward 出去？
cc/ @pm5 bridge 好像怪怪的
好，我下班回家後看一下
好像有點麻煩。可能要明天再來處理
辛苦了 QQ
重開以後在 # 測試恢復了。@mrorz 再試試看吧
工程方向仍是把 url-resolver 搬出去
2026-01-13
Replied to a thread: 2026-01-09 15:25:34
這篇沒被 message bridge forward 出去？
cc/ @pm5 bridge 好像怪怪的
好，我下班回家後看一下
會議裡提及的訊息： https://cofacts.tw/article/2ug7pirf0tb8b
- 👌1
好像有點麻煩。可能要明天再來處理
2026-01-14
Threads 
- 🚑1
剛剛發現了一個也還蠻猛的
剛剛發現了一個也還蠻猛的
歐歐沒事 是斷句有問題
歐歐沒事 是斷句有問題
我連這算不算 hallucination 都不確定
是這樣沒錯但不是這樣.jpg
是這樣沒錯但不是這樣.jpg
- 😂1
- 🧐1
我連這算不算 hallucination 都不確定
是這樣沒錯但不是這樣.jpg
是這樣沒錯但不是這樣.jpg
辛苦了 QQ
重開以後在 # 測試恢復了。@mrorz 再試試看吧
2026-01-15
好耶感謝
2026-01-16
*Incident Summary: Production Server Memory Exhaustion*
Incident Duration: 2026-01-16 09:35 - 09:45 (TPE)
Status: Resolved
*Timeline*
• 09:35 - Report: User reports production server is down.
• 09:36 - Diagnosis: Investigation revealed server load at 29.43 with Memory (16GB) and Swap (4GB) both 100% full. The largest process was identified as `java` (Elasticsearch).
◦ _Source: SSH `top` command._
• 09:38 - Auto-Recovery: The operating system’s OOM Killer terminated the Elasticsearch process to recover memory. The Docker daemon immediately auto-restarted the `db` container.
◦ _Source: `dmesg` logs & `docker inspect` timestamp (09:38:28)._
• 09:44 - Manual Action: User manually restarted the `url-resolver` container to ensure a fresh state.
◦ _Source: User report & `docker ps` uptime._
• 09:45 - Resolution: Server load stabilized at 1.75. All core services (`db`, `api`, `line-bot-zh`) were confirmed up and running.
◦ _Source: SSH `uptime` & `docker-compose ps`._
*Root Cause*
Memory Exhaustion: The Elasticsearch `java` process consumed over 9GB of RAM, eventually filling both physical memory and swap space. This caused the system to thrash and become unresponsive until the OS forcibly killed the process.
*Final Status*
• Healthy: System load is normal.
• Service Uptime:
◦ `db`: ~10 minutes (Auto-recovered)
◦ `url-resolver`: ~4 minutes (Manually restarted)
• Other services: >2 weeks (Unaffected)
Incident Duration: 2026-01-16 09:35 - 09:45 (TPE)
Status: Resolved
*Timeline*
• 09:35 - Report: User reports production server is down.
• 09:36 - Diagnosis: Investigation revealed server load at 29.43 with Memory (16GB) and Swap (4GB) both 100% full. The largest process was identified as `java` (Elasticsearch).
◦ _Source: SSH `top` command._
• 09:38 - Auto-Recovery: The operating system’s OOM Killer terminated the Elasticsearch process to recover memory. The Docker daemon immediately auto-restarted the `db` container.
◦ _Source: `dmesg` logs & `docker inspect` timestamp (09:38:28)._
• 09:44 - Manual Action: User manually restarted the `url-resolver` container to ensure a fresh state.
◦ _Source: User report & `docker ps` uptime._
• 09:45 - Resolution: Server load stabilized at 1.75. All core services (`db`, `api`, `line-bot-zh`) were confirmed up and running.
◦ _Source: SSH `uptime` & `docker-compose ps`._
*Root Cause*
Memory Exhaustion: The Elasticsearch `java` process consumed over 9GB of RAM, eventually filling both physical memory and swap space. This caused the system to thrash and become unresponsive until the OS forcibly killed the process.
*Final Status*
• Healthy: System load is normal.
• Service Uptime:
◦ `db`: ~10 minutes (Auto-recovered)
◦ `url-resolver`: ~4 minutes (Manually restarted)
• Other services: >2 weeks (Unaffected)
Incident Summary: Production Server Memory Exhaustion
Incident Duration: 2026-01-16 09:35 - 09:45 (TPE) Status: Resolved
*Timeline*
• 09:35 - Report: User reports production server is down.
• 09:36 - Diagnosis: Investigation revealed server load at 29.43 with Memory (16GB) and Swap (4GB) both 100% full. The largest process was identified as `java` (Elasticsearch).
◦ _Source: SSH `top` command._
• 09:38 - Auto-Recovery: The operating system’s OOM Killer terminated the Elasticsearch process to recover memory. The Docker daemon immediately auto-restarted the `db` container.
◦ _Source: `dmesg` logs & `docker inspect` timestamp (09:38:28)._
• 09:44 - Manual Action: User manually restarted the `url-resolver` container to ensure a fresh state.
◦ _Source: User report & `docker ps` uptime._
• 09:45 - Resolution: Server load stabilized at 1.75. All core services (`db`, `api`, `line-bot-zh`) were confirmed up and running.
◦ _Source: SSH `uptime` & `docker-compose ps`._
*Root Cause*
Memory Exhaustion: The Elasticsearch
`java` process consumed over 9GB of RAM, eventually filling both physical memory and swap space. This caused the system to thrash and become unresponsive until the OS forcibly killed the process.Final Status
• Healthy: System load is normal.
• Service Uptime:
◦ `db`: ~10 minutes (Auto-recovered)
◦ `url-resolver`: ~4 minutes (Manually restarted)
• Other services: >2 weeks (Unaffected)
Incident Duration: 2026-01-16 09:35 - 09:45 (TPE) Status: Resolved
*Timeline*
• 09:35 - Report: User reports production server is down.
• 09:36 - Diagnosis: Investigation revealed server load at 29.43 with Memory (16GB) and Swap (4GB) both 100% full. The largest process was identified as `java` (Elasticsearch).
◦ _Source: SSH `top` command._
• 09:38 - Auto-Recovery: The operating system’s OOM Killer terminated the Elasticsearch process to recover memory. The Docker daemon immediately auto-restarted the `db` container.
◦ _Source: `dmesg` logs & `docker inspect` timestamp (09:38:28)._
• 09:44 - Manual Action: User manually restarted the `url-resolver` container to ensure a fresh state.
◦ _Source: User report & `docker ps` uptime._
• 09:45 - Resolution: Server load stabilized at 1.75. All core services (`db`, `api`, `line-bot-zh`) were confirmed up and running.
◦ _Source: SSH `uptime` & `docker-compose ps`._
*Root Cause*
Memory Exhaustion: The Elasticsearch
`java` process consumed over 9GB of RAM, eventually filling both physical memory and swap space. This caused the system to thrash and become unresponsive until the OS forcibly killed the process.Final Status
• Healthy: System load is normal.
• Service Uptime:
◦ `db`: ~10 minutes (Auto-recovered)
◦ `url-resolver`: ~4 minutes (Manually restarted)
• Other services: >2 weeks (Unaffected)
2026-01-17
@chiao3.su has joined the channel
2026-01-18
OCR 好像壞掉幾天了
1 月初開始，圖片和影片全都沒有逐字稿，這樣比對功能應該是壞的
超爛，我的 env-file 裡面有幾個 key 在從 yml 複製出來的時候不小心改到變數名
就只有那幾個被改到，其他都沒有
就只有那幾個被改到，其他都沒有
Fixed on 1/18 17:42
OCR 好像壞掉幾天了
1 月初開始，圖片和影片全都沒有逐字稿，這樣比對功能應該是壞的
超爛，我的 env-file 裡面有幾個 key 在從 yml 複製出來的時候不小心改到變數名
就只有那幾個被改到，其他都沒有
就只有那幾個被改到，其他都沒有
Fixed on 1/18 17:42
1 月初開始，圖片和影片全都沒有逐字稿，這樣比對功能應該是壞的
超爛，我的 env-file 裡面有幾個 key 在從 yml 複製出來的時候不小心改到變數名
就只有那幾個被改到，其他都沒有
就只有那幾個被改到，其他都沒有
Fixed on 1/18 17:42
2026-01-19
@vickylee.g123 has left the channel
2026-01-20
# Cofacts 會議記錄 - [搜尋](<https://cse.google.com/cse?cx=71f4f7ee215d54fe6>)[target=_blank] ## 2026 -
2026-01-22
https://github.com/cofacts/rumors-api/pull/378 test 過囉，可以 review 了
結果 test 是死在 transcript，應該是 Gemini model 默默更新導致，因此就速速做了個 fix 然後 merge 進 master 了。
結果 test 是死在 transcript，應該是 Gemini model 默默更新導致，因此就速速做了個 fix 然後 merge 進 master 了。
Test 又不過了，因為我把 test 改成 integration test 來抓問題 ._.
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
https://github.com/cofacts/rumors-api/pull/378 test 過囉，可以 review 了
結果 test 是死在 transcript，應該是 Gemini model 默默更新導致，因此就速速做了個 fix 然後 merge 進 master 了。
結果 test 是死在 transcript，應該是 Gemini model 默默更新導致，因此就速速做了個 fix 然後 merge 進 master 了。
Test 又不過了，因為我把 test 改成 integration test 來抓問題 ._.
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
2026-01-24
Replied to a thread: 2026-01-22 01:54:05
Test 又不過了，因為我把 test 改成 integration test 來抓問題 ._.
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
前幾天我注意到 code 寫錯導致圖片 OCR 根本不會動
然後也修正 staging 上 service account 的權限問題，所以現在 staging 上可以正常作業了
不過變成 unit test 不會動了
我想先 merge 然後在 production 補之前的逐字稿，1 月到上禮拜有些關稅相關的圖應該要做成逐字稿的 orz
2026-01-25
目前無法登入，按了登入方式後顯示 Error 1033
Ray ID: 9c3582e4bc57843f • 2026-01-25 05:53:38 UTCCloudflare Tunnel error
Ray ID: 9c3582e4bc57843f • 2026-01-25 05:53:38 UTCCloudflare Tunnel error
FYI 現在可以了
感謝回報
服務自己活過來真是太好了（？
服務自己活過來真是太好了（？
你已經是個成熟的專案了，該學會自己活過來了(?)
FYI 現在可以了
@clhuang224 has joined the channel
2026-01-26
感謝回報
服務自己活過來真是太好了（？
服務自己活過來真是太好了（？
你已經是個成熟的專案了，該學會自己活過來了(?)
2026-01-27
@chewei 您好，我是今天參與討論提供備援電源韌性建議的 Ed Chen。方便幫我開 HackMD 的筆記權限嗎？感謝🙏
@chewei 不好意思，方便開共筆權限嗎？
感謝🤩
API not accessible now
2026-01-28
昨晚的狀況
• url-resolver 沒有牙起來
• DB RAM usage 也是 9GB 左右
• load 也在 2 以下
但是 api 就是很慢才回來
• url-resolver 沒有牙起來
• DB RAM usage 也是 9GB 左右
• load 也在 2 以下
但是 api 就是很慢才回來
*System Overview*
• *Uptime*: 406 days
• *Load Average*: 1.05, 1.02, 1.07 (on *6 Cores*) -> *Healthy CPU Load*
• *Memory*: 16GB Total
• *Used*: ~12GB (Applications) + ~3GB (Cache)
• *Free*: ~340MB
• *Swap*: *100% Used* (4GB / 4GB) -> *CRITICAL*
*Resource Consumers*
Memory
The system is under heavy memory pressure with Swap completely full.
1. *Elasticsearch (`rumors-deploy_db_1`)*: ~7.83GB (50% of Host RAM)
2. *Uptime*: 2 hours (Recently restarted, yet memory filled again).
3. *Heap Setting*: `ES_JAVA_OPTS=-Xms7g -Xmx7g`.
4. *Docker Daemon (`dockerd`)*: *2.3GB (RES)*. This is unusually high for a daemon and is a significant contributor to memory pressure.
5. *Page Cache*: ~3.1GB. usage is normal for Elasticsearch (Lucene indices interally use OS cache), but in a constrained system, this competes with applications.
Swap Analysis
• *Swap Usage*: 100% (4GB).
• *Swappiness*: 10 (Low).
• *Interpretation*: Even with a low preference for swapping (`vm.swappiness=10`), the system was _forced_ to swap out 4GB because physical RAM was completely exhausted.
CPU
1. *API (`rumors-deploy_api_1`)*:
◦ Process `node /srv/www/build/index.js` (PID 14667) was using *74.6% CPU* in `top` snapshot.
◦ Averaged ~3.5% in `docker stats`.
◦ This indicates traffic spikes or a heavy query processing, but keeping overall load low (~1.0).
2. *Elasticsearch*: ~58% of 1 core (in `docker stats`).
• *Uptime*: 406 days
• *Load Average*: 1.05, 1.02, 1.07 (on *6 Cores*) -> *Healthy CPU Load*
• *Memory*: 16GB Total
• *Used*: ~12GB (Applications) + ~3GB (Cache)
• *Free*: ~340MB
• *Swap*: *100% Used* (4GB / 4GB) -> *CRITICAL*
*Resource Consumers*
Memory
The system is under heavy memory pressure with Swap completely full.
1. *Elasticsearch (`rumors-deploy_db_1`)*: ~7.83GB (50% of Host RAM)
2. *Uptime*: 2 hours (Recently restarted, yet memory filled again).
3. *Heap Setting*: `ES_JAVA_OPTS=-Xms7g -Xmx7g`.
4. *Docker Daemon (`dockerd`)*: *2.3GB (RES)*. This is unusually high for a daemon and is a significant contributor to memory pressure.
5. *Page Cache*: ~3.1GB. usage is normal for Elasticsearch (Lucene indices interally use OS cache), but in a constrained system, this competes with applications.
Swap Analysis
• *Swap Usage*: 100% (4GB).
• *Swappiness*: 10 (Low).
• *Interpretation*: Even with a low preference for swapping (`vm.swappiness=10`), the system was _forced_ to swap out 4GB because physical RAM was completely exhausted.
CPU
1. *API (`rumors-deploy_api_1`)*:
◦ Process `node /srv/www/build/index.js` (PID 14667) was using *74.6% CPU* in `top` snapshot.
◦ Averaged ~3.5% in `docker stats`.
◦ This indicates traffic spikes or a heavy query processing, but keeping overall load low (~1.0).
2. *Elasticsearch*: ~58% of 1 core (in `docker stats`).
*Resolution (Implemented 2026-01-28)*
User applied the following limits to prevent OOM Killer from targeting random system processes:
1. *Elasticsearch (`db`)*:
a. *Java Heap*: `7g`.
b. *Status*: Healthy (7.6GB used / 11GB limit).
2. *URL Resolver*:
a. *Status*: Healthy (82MB used).
*Result*:
1. System Swap is still full (expected without host reboot), but apps are now *contained*.
*Post-Reboot Verification (2026-01-28 03:00台北時間)*
The system rebooted successfully at 19:00 UTC.
Current Status
• *Uptime*: 5 minutes
• *Memory*:
◦ *Used*: 7.3GB (Much lower than the previous 12GB + 4GB Swap)
◦ *Free*: 8.3GB
◦ *Swap*: *0B / 4GB* (Completely cleared!)
• *Docker Daemon*: Memory usage is reset and healthy.
• *Service Verification*:
◦ All core services are *Up*.
◦ _Note_: `db` and `url-resolver` required a manual `up -d` right after reboot as they did not auto-start initially, but they are now running stable with the new limits.
Final Configuration
• *Elasticsearch*: Heap 7G
• *Result*: The system now has ~5-8GB of breathing room for OS cache and other services, significantly reducing the risk of a system-wide freeze.
User applied the following limits to prevent OOM Killer from targeting random system processes:
1. *Elasticsearch (`db`)*:
a. *Java Heap*: `7g`.
b. *Status*: Healthy (7.6GB used / 11GB limit).
2. *URL Resolver*:
a. *Status*: Healthy (82MB used).
*Result*:
1. System Swap is still full (expected without host reboot), but apps are now *contained*.
*Post-Reboot Verification (2026-01-28 03:00台北時間)*
The system rebooted successfully at 19:00 UTC.
Current Status
• *Uptime*: 5 minutes
• *Memory*:
◦ *Used*: 7.3GB (Much lower than the previous 12GB + 4GB Swap)
◦ *Free*: 8.3GB
◦ *Swap*: *0B / 4GB* (Completely cleared!)
• *Docker Daemon*: Memory usage is reset and healthy.
• *Service Verification*:
◦ All core services are *Up*.
◦ _Note_: `db` and `url-resolver` required a manual `up -d` right after reboot as they did not auto-start initially, but they are now running stable with the new limits.
Final Configuration
• *Elasticsearch*: Heap 7G
• *Result*: The system now has ~5-8GB of breathing room for OS cache and other services, significantly reducing the risk of a system-wide freeze.
要注意的事情：現在 Linode 上的 Linux 核心沒有開啟 cgroup memory，導致
• docker stats 現在不會顯示各個 container 的 RAM usage
• docker-compose 無法使用 mem_limit (本來我們也沒在用)
總之現在跟昨晚最大的差別就是
• elasticsearch container 的 Java heap space 從 8GB 調降到 7GB
• 重開過機器所以 swap 清空了、docker daemon 用的 RAM usage 也變低了
• docker stats 現在不會顯示各個 container 的 RAM usage
• docker-compose 無法使用 mem_limit (本來我們也沒在用)
總之現在跟昨晚最大的差別就是
• elasticsearch container 的 Java heap space 從 8GB 調降到 7GB
• 重開過機器所以 swap 清空了、docker daemon 用的 RAM usage 也變低了
好的
再請試試看
再請試試看
好的
再請試試看
再請試試看
下週 2/4 (n0 )
有了，謝謝！
請問可以參與 2/4 小聚嗎？想多了解目前 Cofacts 開發 AI agent 的狀況~
歡迎歡迎
不過最近都在弄別的雜事 XD
沒問題，我也去更新一下你們的雜事XD
五月中受邀到清大通識教育中心分享「AI在人文社會創新」，我想分享幾個案例，包括怎麼用 AI 面對充斥 AI 的假訊息環境😂
五月中受邀到清大通識教育中心分享「AI在人文社會創新」，我想分享幾個案例，包括怎麼用 AI 面對充斥 AI 的假訊息環境😂