clickhouse

pm5 16:13:51

@pm5 has joined the channel

ronnywang 16:13:57

@ronnywang has joined the channel

pm5 16:14:02

他有 ngram 的樣子 https://clickhouse.com/docs/en/sql-reference/functions/splitting-merging-functions#ngrams

pm5 16:14:02

他有 ngram 的樣子 https://clickhouse.com/docs/en/sql-reference/functions/splitting-merging-functions#ngrams

pm5 16:18:36

pm5 16:18:36

helloworld_bot 16:36:49

@helloworld_bot has joined the channel

pm5 16:37:23

咦

pm5 16:37:23

咦

pm5 16:39:15

好像可以?

pm5 16:39:15

好像可以?

pm5 16:44:28

這個設定方式也太神秘了。

pm5 16:44:28

這個設定方式也太神秘了。

pm5 16:48:12

...

pm5 16:48:12

...

pm5 16:50:05

...

pm5 16:50:05

...

pm5 16:52:01

.....

pm5 16:52:01

.....

pm5 16:53:32

..

pm5 16:53:32

..

pm5 16:55:15

?

pm5 16:55:15

?

pm5 16:56:11

忽然變成來做這件事了。開了一個 gateway 把 Telegram clickhouse.tw channel <--> g0v Slack clickhouse channel 連起來。

pm5 16:56:11

忽然變成來做這件事了。開了一個 gateway 把 Telegram clickhouse.tw channel <--> g0v Slack #clickhouse 連起來。

@null 16:58:30

跟 ronny 討論到一半，一時興起，沒有先跟大家說。如果有疑慮的話再麻煩跟我說喔。我們是想在 g0v 找一些喜歡玩資料的人一起來。

ronnywang 23:00:19

社會經濟資料庫的資料真的超適合 clickhouse ，我目前處理到 2013 年，資料筆數已經到六千萬筆了，處理到 2023 應該可以破億

ronnywang 23:00:19

社會經濟資料庫的資料真的超適合 clickhouse ，我目前處理到 2013 年，資料筆數已經到六千萬筆了，處理到 2023 應該可以破億

👍 2

tmonk 23:47:48

@felixtypingmonkey has joined the channel

chihao 13:18:14

@chihao has joined the channel

chihao 13:18:35

億！

chihao 13:18:35

億！

pm5 14:36:15

set the channel description: 玩 ClickHouse 資料庫系統的頻道。與 Telegram clickhouse.tw 群組訊息同步。

ronnywang 16:10:19

😂 2 🦕 1

Charlie Wang 11:01:12

@cwang has joined the channel

clkao 11:06:14

@clkao has joined the channel

poga 11:23:24

@poga has joined the channel

paulpengtw 11:30:12

@paulpengtw has joined the channel

lafin 11:31:00

@lafin has joined the channel

Lucky 11:49:45

@f24931154 has joined the channel

Kyle Yu 12:22:08

@chaukwai has joined the channel

jeffery nobody-f 15:08:43

@jeffery.sac has joined the channel

Amos 15:28:47

@amosli.tw has joined the channel

ronnywang 17:41:11

https://lydata.ronny-s3.click/segis.zip
社會經濟資料庫也處理好了， 2.9G ，資料筆數約2億954萬筆

ronnywang 17:41:11

https://lydata.ronny-s3.click/segis.zip
社會經濟資料庫也處理好了， 2.9G ，資料筆數約2億954萬筆

ronnywang 17:41:34

資料都是 csv

ronnywang 17:41:34

資料都是 csv

ronnywang 18:05:04

分成 2464 個 csv 檔，代表 2464 個不同個 table ，table 的名稱和裡面欄位的定義可以參考 categories.jsonl 或 category.csv

ronnywang 18:05:04

分成 2464 個 csv 檔，代表 2464 個不同個 table ，table 的名稱和裡面欄位的定義可以參考 categories.jsonl 或 category.csv

ronnywang 18:38:47

詳細欄位名稱放入 https://g0v.hackmd.io/gGGUBDEXQOKGL_feUpPekg?both

ronnywang 18:38:47

詳細欄位名稱放入 https://g0v.hackmd.io/gGGUBDEXQOKGL_feUpPekg?both

Lynn 00:25:42

@lynn1221 has joined the channel

Carmen 00:56:30

@carmenkuo0628 has joined the channel

weiting Lin (我們的基因體時代) 12:08:31

@weitinglin66 has joined the channel

itingo 21:50:40

@itingfan has joined the channel

yestin chen 10:13:43

@ggshiun has joined the channel

ddio 15:36:44

@ddio has joined the channel

@null 16:03:26

請問這群組是私人群組嗎?!

@null 16:03:45

想詢問有關clickhouse的問題

@null 20:29:40

任何人都可以加入喔！

@null 20:30:07

恩恩 感謝

@null 20:30:34

有個問題想提問

helloworld_bot 20:32:24

File from [g0vbridgebot] @timdbawith comment: 我透過分佈表寫入 200000筆
但在system.query_log 顯示卻是讀取20000筆??

@null 20:59:22

沒有看到 query 本身長怎樣，有點難隔空抓藥耶。

@null 21:02:08

很單純批量insert這樣

@null 21:18:55

-- definition drop table if exists Game_Demo.demo_local on cluster cluster1 sync; CREATE TABLE Game_Demo.demo_local on cluster cluster1 ( TransferID Decimal(22, 0) , DBCreateTime DateTime64(3) ) ENGINE = MergeTree() PARTITION BY toYYYYMMDD(DBCreateTime) ORDER BY (TransferID) SETTINGS index_granularity = 8192; CREATE TABLE Game_Demo.demo ON CLUSTER cluster1 AS Game_Demo.demo_local ENGINE = Distributed('cluster1','Game_Demo','demo_local', rand()); insert into Game_Demo.demo (TransferID,DBCreateTime)VALUES (1,now()) ,(2,now()) ,(3,now()) ,(4,now()) ,(5,now()) ,(6,now()) ,(7,now()) ,(8,now()) ,(9,now()) ,(10,now()); select * from system.query_log ql where type ='QueryFinish' and query_kind ='Insert' order by event_time desc limit 10;

helloworld_bot 21:20:20

File from [g0vbridgebot] @timdba

@null 21:21:24

是因為分布表收到要寫入的筆數? 所以read_rows 才會收到要批次寫入的筆數?

ifengc 07:14:55

@iamifengc has joined the channel

ronnywang 23:37:07

政治獻金的收支資料我補在 https://g0v.hackmd.io/gGGUBDEXQOKGL_feUpPekg
我發現用 duckdb 玩這資料還不錯方便

ronnywang 23:37:07

政治獻金的收支資料我補在 https://g0v.hackmd.io/gGGUBDEXQOKGL_feUpPekg
我發現用 duckdb 玩這資料還不錯方便

clkao 19:41:36

channel 要改名了嗎 😛

順便加一下 dbt modeling

pm5 22:07:54

哇跟 telegram 的 bridge 好像斷了 XD

😮 1

pm5 22:07:54

哇跟 telegram 的 bridge 好像斷了 XD

pm5 08:30:08

誒 dbt model 好像不錯 XD

clkao 10:00:07

this is a great project modeling many open data https://github.com/davidgasquez/datadex

clkao 10:00:23

and using rill for quick local analytics