cbbrowne is currently certified at Master level.

Name: Christopher Browne
Member since: 2000-07-06 15:59:35
Last Login: 2011-04-18 18:34:37

FOAF RDF Share This

Homepage: http://linuxdatabases.info

Notes: Listed on BlogShares

Projects

Articles Posted by cbbrowne

Recent blog entries by cbbrowne

Syndication: RSS 2.0

PGCon 2018 Unconference

I attended the PGCon 2018 Developer Unconference, on May 30, 2018, which had, as always, a goodly mixture of discussions of things in progress.

Schema Deployment

I proposed a discussion of schema deployment methods; it fell just short of attracting enough interest to get a room. A few people asked me in passing about what I’d had in mind; I’ll point at the Github project, Mahout, which indicates the set of “agenda” that I have in the matter.

Mahout is a lightweight tool (e.g. – needs Bash, psql, and the most sophisticated shell thing needed is tsort), which, being PostgreSQL-specific, could readily be extended to support PG-specific tooling such as Slony.

In the documentation for Mahout, I describe my goals for it, and the things I hope it solves more satisfactorily than some of the other tools out there. I’d be particularly keen on getting any input as to useful purposes that point to changes in functionality.

JIT Compilation

– Andres Freund
– Biggest benefit comes in aggregate processing, that is where there is huge computational work
– planning is presently too simplistic
– no cacheing of compiled JIT code
– code is somewhat slow in some cases
– JIT compilation of COPY would probably be helpful
– COPY, cuts out a lot of presently kludgy C code
– Sorts
– hashing for hash aggregates and hash joins, there is already a prototype…
– interesting idea to capture generated code in a table (Jeff)
– either C or LLVM bitcode
– bitcode may be architecture-dependent
– want better EXPLAIN [ANALYZE] output (Teodor)
– better code generation for expression evaluation (Andres)
– Presently no Windows support
– Once you have cacheing, OLTP performance will improve, but we’re certainly not there now
– local cache, initially; eventually a global cache
– LRU cache
– If I generated much the same code last time, can reuse the compiled code
– would move some work from executor to the planner, but this is a pretty deep architectural change for now
– can definitely get degenerate optimization cases; gotta watch out for that
– generated code is way denser than the executor plans, so there are cases of significant improvements in memory usage
– Incremental JIT compilation (do it in background, after starting query, but before execution)
– impact of threading? Worker backends + data marshalling?

Connection pooler – Odyssey

– Multithreaded connection pooler and request router
– Open source release
Yandex/Odyssey @ GitHub
– Multithreaded via worker thread
– each thread arranges authentication and proxying of client-to-server and server-to-client requests
– worker threads share global server connection pools
– SSL/TLS supported
– tracks transaction state
– can emit CANCEL on connections and ROLLBACK of abandoned transactions before returning connection to pool
– Pools defined as pair of DB/User
– Each pool can authenticate separately and have different mode/limit settings
– UUID identifiers for each connection
– Log events and client error responses include the UUID

Monitoring

– Greg Stark
– Splunk with alerts based on log data
– Nice to have things actively exported by PostgreSQL
– Exposing
– Aggregating
– Reporting
– Error reporting
– Log files too chatty, lot of data all together
– Could different data be separated?
– But how about when it needs to be correlated?
– Sensitive data…
– Too much parsing needs to be done
– Loading into DB makes it structured, but recursive problems, and can overload the DB
– Metrics
– start with What You Want Measured…
– Rates of many things
– vacuum activity
– WAL generation
– error rates
– index usages
– violation statistics, rates of rollbacks and errors
– Nice to have…
– pg stats with numbers of violations and contentions
– let the stats collector collect a bit more data
– connection usage statistics
– Some tools
– Jaeger – JaegerTracing
Zipkin – distributed tracing system to find latency issues in microservice architectures
Opentracing – vender neutral APIs and instrumentation for distributed tracing
– Can there be a “pg stat user table” indicating bloat information?

Query Optimization with Partitioned Tables

– Planned improvements in PG11
– Partition wise pairs
– Partition wise aggregation
– Partition pruning
– Planning time
– Runtime

TDE – Transparent Data Encryption

– Inshung Moon
– Buffer level encryption/decryption
– Per table encryption
– Perhaps should be per-column???
– 2-tier encryption key management
– Working with external key management services (KMS)
– WAL encryption
– only doing encryption on parts other than header
– Nice to have it on LOB (large objects API) too, but no easy way…
– Log file data needs to be encrypted before submission to destinations

Concerns

– Encryption of indexes is troublesome
– You lose the usefulness of ordering of disk
– Table added with a per-table/column private key
– What if some data seems to be exposed? Need to generate new key
and rewrite? This would be arbitrarily expensive…
– Changing master key is easy, as long as the function for
generating the private symmetric (per-table key) is symmetric

Threat model

– Translucent Databases
– Peter Wayner
– Order preserving encryption
– Agrawal encryption scheme
Order Preserving Encryption for Numeric Data, by Agrawal, Kernan, Srikant, Xu
IBM Almaden
SIGMOD paper

Syndicated 2018-06-11 16:47:54 from linuxdatabases.info

Speaking on PostgreSQL

I’m doing a talk on Sept 13th at GTALUG GTALUG Sept 2016 where I’ll be talking on things new in PostgreSQL 9.5 (released in January 2016) and upcoming in 9.6.

Here’s a copy of the material…
PostgreSQL New Stuff

Syndicated 2016-09-06 15:37:25 from linuxdatabases.info

Spamalicious times

Hmmph. Google sent me a “nastygram” indicating that one of my blog entries had something suggestive of content injection.

I poked around, and it was by no means evident that it was really so. The one suspicious posting was http://linuxdatabases.info/blog/?p=99 which legitimately has some stuff that looks like labels, as it contains a bunch of sample SQL code. I’m suspicious that they’re accounting that as being evil…

But it pointed me at a couple of mostly-irritating things…

  1. I haven’t generated a blog entry since 2013. Well, I’m not actually hugely worried about that.
  2. I reviewed proposed response posts, since, probably about 2013. Wow, oh wow, was that ever spam-filled. Literally several thousand attempts to get me to publish various and sundry advertising links. It’s seriously a pain to get rid of them all, as I could only trim out about 150 at a time. And hopefully there weren’t many “real” proposed postings; it’s almost certain I’ll have thrown those away. (Of course, proposed postings about things I said in 2013… How relevant could it still be???)

Syndicated 2016-08-15 20:44:52 from linuxdatabases.info

Nexus 7 on CyanogenMod

At last…

I had been lazy, leaving all alone.

In February, I figured I was heading off for a chunk of the month on a cruise, hence wanting tablet for multimedia, but without network, so it was timely not to spend time fiddling with configuration with possible risk of mussing such up.

Alas, the OTA upgrade to JellyBean did a certain chunk of mussing…  It busted SuperUser access, thereby breaking Titanium Backup.  No backups went properly since :-( .

So, today seemed right timing.  I wanted backup, and needed root, the latter looking like a fight.  Ah well, go for gusto, see what we get without it…

I had to upgrade adb to support latest Android…   Got Clockwork Recovery in place, and zip files for CM10.1 and Google Apps…

The last backup was Feb 16, but happily the files still remained after fresh CM10.1 installation, so I could do a good chunk of recovery of apps, and in plenty of cases, this was basically network configuration, so apps would update their own data upon startup.  Sweet!

Superuser is nicely integrated into CM10, also sweet, no extra installation process.

I’ll need to reconfigure the launcher, due to the shift from ADW (I had a license) to built-in Trebuchet on CM10, but that seems like the “worst” irritation, and one I can well live with.

I’m not sure I can readily identify big differences between stock Android and CM10, but there are nice small creature comforts my CM10 phone has gotten me used to, like a quick “turn on/off WiFi” directly on notification screens.  Small but I like it.

Syndicated 2013-03-09 00:18:21 from linuxdatabases.info

Mailman subscriber lists

As part of “due diligence” for some mailing lists I am involved with (for Slony, see slony-backups ), I discovered the need to dump out Mailman mailing list subscribers.

There is a script to do this, written in Python, mentioned on the Mailman wiki, accessible as mailman-subscribers.py

I’d kind of rather have something a bit more version-tracked, so I poked around at GitHub, and found larsks / mailman-subscribers

That was a little out of date; the last code was from a couple of years ago, so I forked, updated to the latest, and suggested that “larsks” pull it, which he did, quite quickly.

The “kudos” bit is that I noticed a bit of a blemish, in that the mailing list password was required to be on the command line, thereby making it visible to anyone with access to /usr/bin/ps on one’s system. I submitted a feature request, and Lars was so kind as to have this feature added so quickly that by the time I had the prototype of my Slony “subscriber backup” script working, I immediately needed to change it to make use of the lovely new password-in-file feature. Nice!

Syndicated 2013-02-27 18:32:42 from linuxdatabases.info

30 older entries...

 

cbbrowne certified others as follows:

  • cbbrowne certified graydon as Master
  • cbbrowne certified jkh as Master
  • cbbrowne certified jallison as Master
  • cbbrowne certified cbbrowne as Journeyer
  • cbbrowne certified craigbro as Journeyer
  • cbbrowne certified argent as Master
  • cbbrowne certified cmm as Journeyer
  • cbbrowne certified moshez as Journeyer
  • cbbrowne certified ralsina as Journeyer
  • cbbrowne certified jrennie as Apprentice
  • cbbrowne certified wichert as Master
  • cbbrowne certified hpa as Master
  • cbbrowne certified alan as Master
  • cbbrowne certified BrucePerens as Master
  • cbbrowne certified miguel as Master
  • cbbrowne certified blizzard as Journeyer
  • cbbrowne certified jbuck as Master
  • cbbrowne certified corbet as Journeyer
  • cbbrowne certified cdegroot as Journeyer
  • cbbrowne certified jg as Master
  • cbbrowne certified jwz as Master
  • cbbrowne certified johnsonm as Master
  • cbbrowne certified kojima as Journeyer
  • cbbrowne certified sopwith as Master
  • cbbrowne certified eMBee as Journeyer
  • cbbrowne certified dneighbors as Journeyer
  • cbbrowne certified kmacleod as Journeyer
  • cbbrowne certified mjs as Master
  • cbbrowne certified Darin as Master
  • cbbrowne certified phk as Master
  • cbbrowne certified deven as Apprentice
  • cbbrowne certified edw as Apprentice
  • cbbrowne certified mbp as Journeyer
  • cbbrowne certified kmself as Apprentice
  • cbbrowne certified wurtel as Journeyer
  • cbbrowne certified jhasler as Journeyer
  • cbbrowne certified esr as Journeyer
  • cbbrowne certified chalst as Apprentice
  • cbbrowne certified eldamitri as Apprentice
  • cbbrowne certified jbs as Master
  • cbbrowne certified linas as Master
  • cbbrowne certified rjain as Journeyer
  • cbbrowne certified kendall as Apprentice
  • cbbrowne certified pphaneuf as Journeyer

Others have certified cbbrowne as follows:

  • cbbrowne certified cbbrowne as Journeyer
  • cmm certified cbbrowne as Journeyer
  • eMBee certified cbbrowne as Journeyer
  • dneighbors certified cbbrowne as Journeyer
  • jrennie certified cbbrowne as Journeyer
  • mwh certified cbbrowne as Journeyer
  • mjs certified cbbrowne as Journeyer
  • kmself certified cbbrowne as Journeyer
  • jonkare certified cbbrowne as Journeyer
  • mbp certified cbbrowne as Journeyer
  • chalst certified cbbrowne as Journeyer
  • jhasler certified cbbrowne as Journeyer
  • jbs certified cbbrowne as Journeyer
  • eldamitri certified cbbrowne as Journeyer
  • tbmoore certified cbbrowne as Journeyer
  • sej certified cbbrowne as Journeyer
  • linas certified cbbrowne as Master
  • rjain certified cbbrowne as Journeyer
  • auspex certified cbbrowne as Journeyer
  • grant certified cbbrowne as Journeyer
  • pphaneuf certified cbbrowne as Journeyer
  • kendall certified cbbrowne as Journeyer
  • wspace certified cbbrowne as Journeyer
  • pencechp certified cbbrowne as Master
  • sye certified cbbrowne as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page