pg_basebackup: error: could not initiate base backup: ERROR: could not set compression worker count to 4: Unsupported parameter
Friday, November 04, 2022
pg_basebackup could not set compression worker count - unsupported parameter
PostgreSQL 15 hasn't been released for very long, but some people have already been confused by the following error message:
Monday, May 02, 2022
Parallel Server-Side Backup Compression
I decided to do a little more research on the performance of server-side backup compression, which will be a new feature in PostgreSQL 15 unless, for some reason, the changes need to be reverted prior to release time. The network link I used for my previous testing was, as I mentioned, rather slow, and handicapped by both a VPN link and an SSH tunnel. Furthermore, I was testing using pgbench data, which is extremely compressible. In addition, at the time I did those tests, we had added support for LZ4 compression, but we had not yet added support for Zstandard compression. Now, however, we not only have Zstandard as an option, but it is possible to use the library's multi-threading capabilities. So, I wanted to find out how things would work out on a faster network link, with a better test data set, and with all the compression algorithms that we now have available.
Friday, February 11, 2022
Server-Side LZ4 Backup Compression
I have been working with my colleagues Tushar Ahuja, Jeevan Ladhe, and Dipesh Pandit to make some improvements to pg_basebackup for version 15. A lot of that work has felt a bit like boring but necessary refactoring, and it's easy to find yourself wondering whether it will really do anybody any good. I was feeling optimistic after today's commits, so I decide to give it a try.
Tuesday, January 18, 2022
Surviving Without A Superuser - Part Two
If PostgreSQL had the ability to give to a privileged non-superuser the right to administer objects belonging to some designated group of superusers just as if the privileged account were superuser, it would get us much closer to a world in which the database can be effectively administered by a non-superuser. A highly privileged user - let's call him sauron - could be given the right to administer tables, schemas, functions, procedures, and a variety of other objects owned by dependent users witchking and khamul just as if sauron were superuser. sauron might indeed feel himself to be virtually a superuser, at least within his own domain, as long as he didn't spend too much time thinking about the users over which he had not been given administrative rights. However, sauron might notice a few irksome limitations.
Wednesday, January 12, 2022
Who Contributed to PostgreSQL Development in 2020 and 2021?
I have done a few previous blog posts on who has contributed to PostgreSQL, but I did not do one last year. A couple people mentioned to me that they missed it, so I decided to do one this year, and I decided to gather statistics, using basically the same methodology that I have in the past, for both 2020 and 2021. As always, it is important to remember, first, that many people contribute to the project in ways that these statistics don't capture, and second, that the statistics themselves are prone to mistakes (since there is a bunch of manual work involved) and bias (since each commit is attributed to the first author for lack of knowledge of the relative contributions of the various authors). As usual, I have posted a dump of the database I used to generate this in case anyone wants to check it over for goofs or use it for any other purpose.
I calculate that there were 176 people who were the principal author of at least one PostgreSQL commit in 2020 and 182 such people in 2021. In 2020, 13 people contributed 66% of the lines of new code, and 35 people contributed 90% of the lines of new code. In 2021, these numbers were 14 and 41 respectively. In 2020, there were a total of 2181 commits from 26 committers, and in 2021, there were 2269 commits from 28 committers. In each year, about 5 committers committed about two thirds of the non-self-authored patches, with Tom Lane leading the pack in both years.
Here are the top 35 contributors by lines of new code contributed in 2020. Asterisks indicate non-committers. Note that some of these people are committers now but were not committers at the time.
# | author | lines | pct_lines | commits
----+---------------------------+-------+-----------+---------
1 | Tom Lane | 65203 | 25.95 | 436
2 | Peter Eisentraut | 28771 | 11.45 | 229
3 | Paul Jungwirth [*] | 10723 | 4.27 | 2
4 | Heikki Linnakangas | 8293 | 3.30 | 31
5 | Robert Haas | 7831 | 3.12 | 37
6 | Tomas Vondra | 7461 | 2.97 | 32
7 | Andres Freund | 6614 | 2.63 | 59
8 | John Naylor | 6060 | 2.41 | 14
9 | Michael Paquier | 5744 | 2.29 | 103
10 | Ashutosh Bapat [*] | 5515 | 2.20 | 7
11 | Bruce Momjian | 5077 | 2.02 | 121
12 | Nikita Glukhov [*] | 4982 | 1.98 | 7
13 | Thomas Munro | 4820 | 1.92 | 57
14 | James Coleman [*] | 4798 | 1.91 | 16
15 | Mark Dilger [*] | 4779 | 1.90 | 12
16 | Kyotaro Horiguchi [*] | 4775 | 1.90 | 35
17 | Peter Geoghegan | 3696 | 1.47 | 109
18 | Amit Langote [*] | 3650 | 1.45 | 28
19 | Anastasia Lubennikova [*] | 3611 | 1.44 | 4
20 | Jeff Davis | 3408 | 1.36 | 40
21 | Alvaro Herrera | 3073 | 1.22 | 75
22 | Pavel Stehule [*] | 2752 | 1.10 | 4
23 | Julien Rouhaud [*] | 2656 | 1.06 | 19
24 | Alexander Korotkov | 2613 | 1.04 | 33
25 | Masahiko Sawada [*] | 2540 | 1.01 | 20
26 | Dilip Kumar [*] | 2306 | 0.92 | 15
27 | Justin Pryzby [*] | 2222 | 0.88 | 53
28 | Fujii Masao | 2139 | 0.85 | 51
29 | Daniel Gustafsson | 1835 | 0.73 | 54
30 | Corey Huinker [*] | 1777 | 0.71 | 1
31 | Dmitry Dolgov [*] | 1628 | 0.65 | 3
32 | Dean Rasheed | 1512 | 0.60 | 9
33 | David Rowley | 1382 | 0.55 | 35
34 | Vik Fearing [*] | 1285 | 0.51 | 5
35 | Karl Pinc [*] | 1278 | 0.51 | 1
And here are the top 41 contributors by lines of new code contributed in 2021.
# | ?column? | lines | pct_lines | commits
----+------------------------------+-------+-----------+---------
1 | Tom Lane | 66210 | 26.09 | 438
2 | Tomas Vondra | 15357 | 6.05 | 50
3 | Dagfinn Ilmari Mannsåker [*] | 14715 | 5.80 | 10
4 | Peter Eisentraut | 12976 | 5.11 | 214
5 | Robert Haas | 7035 | 2.77 | 46
6 | Bruce Momjian | 7010 | 2.76 | 58
7 | Peter Geoghegan | 6889 | 2.71 | 91
8 | Amit Langote [*] | 6859 | 2.70 | 24
9 | Heikki Linnakangas | 6706 | 2.64 | 38
10 | Mark Dilger [*] | 6203 | 2.44 | 23
11 | David Rowley | 5848 | 2.30 | 50
12 | Alvaro Herrera | 5582 | 2.20 | 79
13 | Andres Freund | 5288 | 2.08 | 53
14 | Michael Paquier | 5057 | 1.99 | 127
15 | Thomas Munro | 4356 | 1.72 | 78
16 | Peter Smith [*] | 4194 | 1.65 | 29
17 | Vignesh C [*] | 3886 | 1.53 | 19
18 | Dilip Kumar [*] | 3496 | 1.38 | 19
19 | Craig Ringer [*] | 3070 | 1.21 | 6
20 | Masahiko Sawada [*] | 2879 | 1.13 | 32
21 | Andrew Dunstan | 2461 | 0.97 | 48
22 | Bharath Rupireddy [*] | 2336 | 0.92 | 41
23 | Daniel Gustafsson | 2127 | 0.84 | 38
24 | Justin Pryzby [*] | 2087 | 0.82 | 51
25 | Hayato Kuroda [*] | 2080 | 0.82 | 5
26 | Ajin Cherian [*] | 2021 | 0.80 | 10
27 | Kyotaro Horiguchi [*] | 1896 | 0.75 | 28
28 | John Naylor | 1814 | 0.71 | 20
29 | Greg Nancarrow [*] | 1745 | 0.69 | 12
30 | Julien Rouhaud [*] | 1707 | 0.67 | 16
31 | Edmund Horner [*] | 1654 | 0.65 | 1
32 | Noah Misch | 1605 | 0.63 | 30
33 | Dmitry Dolgov [*] | 1536 | 0.61 | 3
34 | Amit Kapila | 1434 | 0.57 | 21
35 | Fabien Coelho [*] | 1398 | 0.55 | 7
36 | Gilles Darold [*] | 1338 | 0.53 | 1
37 | Jacob Champion [*] | 1247 | 0.49 | 7
38 | Andrey Borodin [*] | 1224 | 0.48 | 6
39 | Dean Rasheed | 1131 | 0.45 | 12
40 | Nathan Bossart [*] | 1113 | 0.44 | 15
41 | Alexander Korotkov | 1049 | 0.41 | 17
Next let's look at which committers did the most work committing patches that they did not themselves write. Here is the data for 2020.
# | committer | lines | pct_lines | commits
----+--------------------+-------+-----------+---------
1 | Tom Lane | 18728 | 19.25 | 132
2 | Alexander Korotkov | 17043 | 17.51 | 16
3 | Michael Paquier | 11932 | 12.26 | 143
4 | Amit Kapila | 10327 | 10.61 | 74
5 | Alvaro Herrera | 6442 | 6.62 | 51
6 | Etsuro Fujita | 5393 | 5.54 | 2
7 | Tomas Vondra | 5008 | 5.15 | 17
8 | Peter Geoghegan | 3823 | 3.93 | 6
9 | Robert Haas | 3468 | 3.56 | 12
10 | Peter Eisentraut | 3051 | 3.14 | 32
11 | Noah Misch | 3046 | 3.13 | 5
12 | Fujii Masao | 2646 | 2.72 | 61
13 | Heikki Linnakangas | 1954 | 2.01 | 25
14 | Dean Rasheed | 1458 | 1.50 | 2
15 | David Rowley | 583 | 0.60 | 8
16 | Andres Freund | 516 | 0.53 | 6
17 | Thomas Munro | 516 | 0.53 | 13
18 | Andrew Gierth | 374 | 0.38 | 3
19 | Magnus Hagander | 257 | 0.26 | 18
20 | Michael Meskes | 201 | 0.21 | 1
21 | Bruce Momjian | 182 | 0.19 | 12
22 | Stephen Frost | 134 | 0.14 | 2
23 | Jeff Davis | 130 | 0.13 | 2
24 | Andrew Dunstan | 97 | 0.10 | 4
And here are the non-self-authored commits for 2021.
# | committer | lines | pct_lines | commits
----+--------------------+-------+-----------+---------
1 | Tom Lane | 25692 | 26.83 | 111
2 | Amit Kapila | 16269 | 16.99 | 105
3 | Robert Haas | 8886 | 9.28 | 30
4 | Michael Paquier | 8343 | 8.71 | 163
5 | Peter Eisentraut | 6787 | 7.09 | 35
6 | Alvaro Herrera | 6103 | 6.37 | 44
7 | Fujii Masao | 4379 | 4.57 | 70
8 | Tomas Vondra | 3277 | 3.42 | 19
9 | David Rowley | 2807 | 2.93 | 20
10 | Etsuro Fujita | 2216 | 2.31 | 5
11 | Michael Meskes | 1653 | 1.73 | 2
12 | Alexander Korotkov | 1540 | 1.61 | 5
13 | Thomas Munro | 1469 | 1.53 | 16
14 | Bruce Momjian | 1283 | 1.34 | 9
15 | Daniel Gustafsson | 1030 | 1.08 | 25
16 | Peter Geoghegan | 960 | 1.00 | 11
17 | Magnus Hagander | 823 | 0.86 | 12
18 | Noah Misch | 578 | 0.60 | 8
19 | Heikki Linnakangas | 553 | 0.58 | 7
20 | Stephen Frost | 291 | 0.30 | 3
21 | Dean Rasheed | 260 | 0.27 | 2
22 | John Naylor | 253 | 0.26 | 6
23 | Andrew Dunstan | 110 | 0.11 | 4
24 | Jeff Davis | 103 | 0.11 | 2
25 | Andres Freund | 76 | 0.08 | 3
26 | Joe Conway | 10 | 0.01 | 1
Finally, let's look at who sent a lot of emails to pgsql-hackers each year. Many of these people are also prolific patch authors, but some of them are more involved in discussion and review than in actually writing code. Here is everyone who sent at least 100 emails to the list in 2020.
count | name
-------+-----------------------
2539 | Tom Lane
1484 | Justin Pryzby
1389 | Michael Paquier
1156 | Amit Kapila
1068 | Alvaro Herrera
992 | Kyotaro Horiguchi
986 | Andres Freund
869 | Robert Haas
803 | Tomas Vondra
658 | Fujii Masao
620 | Peter Eisentraut
569 | Thomas Munro
530 | Masahiko Sawada
507 | Bruce Momjian
489 | Pavel Stehule
486 | Peter Geoghegan
453 | Julien Rouhaud
445 | Dilip Kumar
368 | Daniel Gustafsson
356 | David Rowley
355 | Stephen Frost
332 | Amit Langote
298 | Bharath Rupireddy
270 | Heikki Linnakangas
245 | James Coleman
235 | Ranier Vilela
214 | Andy Fan
210 | Andrew Dunstan
208 | Mark Dilger
203 | Magnus Hagander
197 | Takayuki Tsunakawa
189 | Alexander Korotkov
189 | Fabien Coelho
185 | David G. Johnston
177 | Konstantin Knizhnik
174 | David Steele
161 | Vignesh C
159 | Ashutosh Bapat
153 | Jeff Davis
152 | Noah Misch
145 | John Naylor
139 | Craig Ringer
130 | Alexey Kondratov
118 | Laurenz Albe
114 | Anastasia Lubennikova
112 | Dmitry Dolgov
110 | Wenjing Zeng
101 | Vik Fearing
And here is the same for 2021.
count | name
-------+-----------------------
2476 | Tom Lane
1385 | Amit Kapila
1378 | Michael Paquier
1178 | Justin Pryzby
1056 | Andres Freund
1056 | Alvaro Herrera
966 | Robert Haas
862 | Bharath Rupireddy
831 | Kyotaro Horiguchi
775 | Tomas Vondra
675 | Masahiko Sawada
648 | Andrew Dunstan
623 | Bruce Momjian
619 | Peter Geoghegan
615 | Dilip Kumar
525 | Fujii Masao
517 | Vignesh C
512 | Thomas Munro
501 | Peter Eisentraut
484 | Daniel Gustafsson
478 | Mark Dilger
460 | Julien Rouhaud
415 | Nathan Bossart
411 | Peter Smith
397 | Pavel Stehule
381 | David Rowley
360 | Stephen Frost
356 | Amit Langote
354 | Zhijie Hou
348 | Greg Nancarrow
291 | Zhihong Yu
242 | Joel Jacobson
235 | Heikki Linnakangas
231 | John Naylor
216 | Jacob Champion
199 | Fabien Coelho
196 | Takayuki Tsunakawa
196 | Magnus Hagander
183 | Ranier Vilela
177 | Noah Misch
175 | Osumi Takamichi
168 | Japin Li
155 | Yugo Nagata
153 | Jeff Davis
152 | Haiying Tang
149 | Amul Sul
147 | Andrey Borodin
140 | Ajin Cherian
138 | David Steele
132 | Laurenz Albe
128 | David G. Johnston
122 | Ronan Dunklau
111 | Simon Riggs
108 | Matthias Van De Meent
107 | Euler Taveira
106 | Andy Fan
103 | Dean Rasheed
I apologize if these email numbers are not completely accurate, and especially to our contributors from China and Japan. Some people posted under multiple names, or using names not written in the character set with which I am most familiar, and I did my best to figure out which posts were actually from the same person and how to best render that person's name. However, I suspect that I have not been able to be entirely consistent about the ordering of family names as opposed to given names, and I may have made some other mistakes as well. I apologize for and regret my errors.
As always, thanks to all who have contributed in any way, whether these lists capture that contribution or not!
Subscribe to:
Posts (Atom)