Tuesday 1 April 2014

org.apache.thrift.transport.TTransportException while dropping or updating a keyspace with Cassandra 1.2

Cassandra offers great flexibility and power to the developer like no database before it IMHO. That also means the ability to screw it up. Recently I migrated Cassandra to a new datacenter where to original set up was spread over 2 datacenters. This meant temporarily adding a 3rd datacenter, then tearing down one of the old ones.

This meant the developers had to adjust their deployment scripts to include the new datacenter, eg:

create keyspace MyKS
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC1 : 3, DC2 : 3}
and durable_writes = true;

became this:

create keyspace MyKS
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC1 : 3, DC2 : 3, DC3 : 3}
and durable_writes = true;

This meant updating the cassandra-topology.properties file on each node and DC3 added.

Migration complete, DC3 was renamed to DC1 and the original DC1 was taken out in the topology files. Unfortunately a number of deployment scripts did not get reverted to the original 2 DC set up and continued to reference DC3 in drop and create scripts.

This mostly worked harmlessly under 1.0, but a recent upgrade to 1.2 became more problematic. Developers complained that they could no longer drop keyspaces with errors such as:

org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.cassandra.thrift.Cassandra$Client.recv_system_drop_keyspace(Cassandra.java:1437)
at org.apache.cassandra.thrift.Cassandra$Client.system_drop_keyspace(Cassandra.java:1424)
at org.apache.cassandra.cli.CliClient.executeDelKeySpace(CliClient.java:1364)
at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:249)
at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:213)
at org.apache.cassandra.cli.CliMain.evaluateFileStatements(CliMain.java:393)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:272)

Under v1.0 about 1 in 5-10 attempts to drop a keyspace would randomly fail even though DC3 was still referenced in the scripts. Under v1.2, it was consistently doing so.

The developers updated their script and references to DC3 removed. I then attempted to update one of the keyspaces manually (using the cassandra-cli) to rid it of DC3 but was left with the solitary error message:


org.apache.thrift.transport.TTransportException

Stubborn!

I then snuck the DC3 configuration back into the cassandra-topology.properties file on the node which I was working. Even though the DC3 nodes no longer existed, that did the trick! DC3 was then removed by updating all affected keyspaces using the cli. Now the deployment scripts drop/create keyspaces without issue.

Hope that is helpful to someone out there!

UPDATE 3 April 2014

Unfortunately I spoke to soon. Although the cluster was seemingly ok (according to nodetool ring/info/describecluster), there were errors in the log of the nature of:

(on the node where I made the cassandra topology update)

java.lang.RuntimeException: java.io.IOException: Connection reset by peer
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:59)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:30)
at org.apache.cassandra.db.ColumnFamilySerializer.serialize(ColumnFamilySerializer.java:73)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:392)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:377)
at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:255)
at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:201)
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:149)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:59)
at java.nio.channels.Channels.writeFully(Channels.java:81)
at java.nio.channels.Channels.access$000(Channels.java:47)
at java.nio.channels.Channels$1.write(Channels.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.xerial.snappy.SnappyOutputStream.dump(SnappyOutputStream.java:297)
at org.xerial.snappy.SnappyOutputStream.rawWrite(SnappyOutputStream.java:244)
at org.xerial.snappy.SnappyOutputStream.write(SnappyOutputStream.java:99)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.cassandra.utils.ByteBufferUtil.write(ByteBufferUtil.java:328)
at org.apache.cassandra.utils.ByteBufferUtil.writeWithLength(ByteBufferUtil.java:315)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:55)
... 8 more

(on other nodes)
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=eb68069e-20b7-394c-9a3f-a727d45ec594

I then noticed that root had ownership of the cassandra-topology.properties file. So I quickly changed this, but it made no difference. Then I restarted the instance and that made no difference either.

Regarding the error on the other node, it looks like a schema mismatch, but running 'nodetool describecluster' indicated that the schema was consistent across all nodes. Nonetheless I decided to shutdown this node, clear out the system keyspace directory and restart. This produced an interesting result. Initially I could see the same UnknownColumnFamilyException error appearing, but once it had rebuild the system keyspace directory the error went away. Same goes for the original node oddly enough.










No comments:

Post a Comment