What's the standard way to encrypt a file with a public key in Java?

Question

I'm writing a client app in Java that sends encrypted data back to the server. All clients have my RSA public key. Is there a standard way to encrypt a given file with it to send it back to the server?

From websites such as this one, I gather that usually, a symmetric key is generated to encrypt the actual file, and this key is then encrypted with the RSA public key and sent along the file. Is there a standard way in Java or a standard file format (without additional libs) to package the encrypted symmetric key and the symmetrically encrypted file?

Of course, I could design my own file format, but I'd rather stick to standards if possible.

BouncyCastle maybe? Its the only library I know of that does encryption — TheLQ, Jan 04 '11 at 15:11
What is your underlying goal and threat model? E.g. do you also want to authenticate where the data came from, or are you vulnerable to a MITM attack? Have you considered SSL and then encrypting the file on the server side if desired? — nealmcb, Jan 04 '11 at 15:53
+1 to @nealmcb, it shouldn't be assumed that file encryption is necessarily the way to go. — AviD, Jan 04 '11 at 18:58
The client app saves user activity logs whose content should be published to a server unaltered, even if the user has read/write access to the saved files. Moreover, the user should not be able to inspect the contents (the files are exercise, progress, and error reports from a quiz application). — Jean-Philippe Pellet, Jan 04 '11 at 19:51
@JPP, ah see - all this file encryption stuff won't help you now. — AviD, Jan 09 '11 at 00:17
@Avid, I don't understand how encryption doesn't help me — it seems to do what I want. Save the files encrypted with my RSA public key (client cannot decrypt back) and send them to my server when time comes. If the client alters the file, it is then immediately invalid because I can't decrypt it any more, which is exactly what I want. Where's the catch? — Jean-Philippe Pellet, Jan 12 '11 at 11:32

nealmcb · Accepted Answer · 2011-01-04T17:23:37.090

One common, standard encryption format is Cryptographic Message Syntax (CMS, evolved from PKCS#7), and e.g. BouncyCastle supports it. It allows for signatures also, and is the basis, e.g., for S/MIME mail message security. It may be overkill, but is widely supported.

Example code is provided by Bouncy Castle. For CMS, see org.bouncycastle.cms.test, probably the Enveloped Data examples.

Another option is the XML Encryption format, standardized by W3C. For Java examples, see e.g. Exploring XML Encryption, Part 1 or do it with XSS4J: Implementing XML Encryption in Java

score 3 · Answer 2 · answered Jan 07 '11 at 05:22

3

The simplest and most surefire way of file encryption is to shell out and invoke gpg or pgp to encrypt it. I know it sounds inelegant, but in my experience reviewing a lot of crypto code, this is the simplest way to ensure that your solution is secure, without having to know a lot about cryptography. If you want to do it yourself, you will need to learn more about how cryptography works, and the risk of a security problem increases.

Make sure there is authenticity protection for the encrypted data (e.g., encrypt and then sign/MAC): you will need both confidentiality and authenticity protection. A very common error is to encrypt but fail to sign; this error can introduce subtle security problems. (If you are not a cryptographer, I realize this may not make any sense to you. I realize this is by no means obvious, but I am a cryptographer. If you are familiar with IND-CCA2 and INT-CTXT, you will understand why I am recommending this. If you're not familiar with those concepts, you can read about them if you want, or you can take my word on it. Hey, crypto is tricky stuff; what can I say.) For instance, the website you link to makes this mistake; I should not rely upon that website for advice about secure encryption. The strawman approach you mention in your question also suffers from this flaw, and it looks like some of the proposed answers here also have a similar shortcoming.

An alternative approach, which is a very good one if the client and server are connected in real time, is to open a TLS-encrypted channel between the client and the server and send the data over that channel. Make sure that the client verifies the public key / cert of the server. If you want to authenticate clients, give clients a client cert and make sure that the server requires the client to provide a client cert and validates that cert.

answered Jan 07 '11 at 05:22

D.W.

99,525
33
275
596

4
Adds requirement that the gpg/pgp tool is installed on the client box. 2. You now have a new problem: you need to ensure the integrity of a tool you're calling via exec().

Jan 07 '11 at 09:47

1

@Graham, yes, this requires gpg/pgp. No, this is not a new problem: you already have to ensure the integrity of your code, so use the exact same mechanism for gpg/pgp. In practice integrity is not hard. Note: I didn't say my solution has no drawbacks, I said it was the simplest way to get good security. If you have a simpler way to do it securely, feel free to post it. But be warned in advance that there are many tricky details. – D.W. Jan 08 '11 at 00:39

1

@D.W. but you have a race condition. if(pgpExecutable.isValid()) { exec(pgpExecutable.path(), ...) – Jan 08 '11 at 10:31

1

@Graham, you are simply wrong. That is not the solution I was proposing, nor does my solution have a race condition or any other such problem. Sounds like you're a bit confused about the threat model and how software is deployed to end machines. – D.W. Jan 09 '11 at 01:05

It does: it has the one I demonstrated in my comment. – Jan 09 '11 at 16:46

1

@Graham, again, you are wrong. As I already stated, you are attacking a strawman. I am not proposing what you think I am proposing. – D.W. Jan 12 '11 at 02:41

1

@D.W. in that case please make your answer more specific. It sure looks like you are, and I'm sure most readers will find "specific example of a problem"/"you're wrong and I can't be bothered to explain why" boring after a couple of rounds ;) – Jan 12 '11 at 11:02

@Graham , Yes, I agree that if the pgp tool that the Java program uses is compromised, then we have a problem. However (and I think @D.W. will agree with me), this is not a "new problem", compared to doing a functionally equivalent encryption "internally" in Java to avoid dependencies. Any cracker that can corrupt the pgp tool can just as easily corrupt the functionally equivalent code in the Java application. – David Cary Feb 28 '11 at 07:02

1

@David Cary, yes, I agree. (To elaborate on my earlier statement: "you already have to ensure the integrity of your [Java] code, so use the exact same mechanism for [ensuring the integrity of] gpg/pgp".) Oh well, that's one of the downsides of the Stack Exchange format on topics like cryptography; upvotes aren't always correlated to the best technical advice. – D.W. Feb 28 '11 at 09:17

score 3 · Answer 3 · answered Jan 09 '11 at 00:30

@JPP, This comment of yours makes me think you're asking the wrong question:

The client app saves user activity logs whose content should be published to a server unaltered, even if the user has read/write access to the saved files. Moreover, the user should not be able to inspect the contents (the files are exercise, progress, and error reports from a quiz application).

If the client app writes the log files, and encrypts them - that means the client app has access to the encryption key, whatever that may be.
However, you're trying to prevent the user of the client app from reading the files!
That's just not going to work, because:

The user has access to anything the client app has access to - including the file contents, and the encryption key...
The user has control of anything running in his own process - i.e. he can even attach a debugger to the client app - so don't bother trying to get smart and figure out a way to obfuscate the encryption key.

To me, your problem sound more like one of access control - how can I write a log file from a client app, without the user having access to the contents?
One way would be to build a local service (e.g. Windows Service, on Windows) that runs in a different user context; the client app would talk to the service, and the service would then write the contents to the file in a protected location that regular users cannot reach.
Another alternative is, as @D.W. suggested, send it directly to the server immediately, and not write to file any of the contents.

The problem with both these solutions, is as I previously noted: The user has control of anything in his process, and can block or modify any requests sent, either to the server or the local Windows service.

I don't see the problem if I use my public certificate to encrypt what the client writes to disk, and then use my private key to decrypt it once I have received it. — Jean-Philippe Pellet, Jan 12 '11 at 11:25
@JPP, even assuming that the private key is not available on the client machine, but rather only on the server, note also point #2 above, wrt e.g. attaching a debugger - if the malicious user is determined to subvert the process, he will be able to. Thats not to say that encrypting with the standalone public key wont add value - it does raise the bar somewhat, but just so long as you realize thats ALL it does, and it can be gotten around. — AviD, Jan 12 '11 at 11:30
Thanks for the explanation. Sure, attaching a debugger and directly modifying the byte array before it's encrypted is possible. But what's the alternative? Discard all user data generated this way if he's not able to communicate with the server right now? And even then: I can also attach the debugger and modify the byte array before the client sends it to the server, or before it gets sent to a secure Windows Service, so the situation's not better. Or am I missing something? — Jean-Philippe Pellet, Jan 12 '11 at 11:39
No, @JPP, not missing something - except for the very basic fact that you cannot rely on the client. Overall, and given the constraints, the encryption as you laid it is probably the best, but take it for what its worth. It will prevent tampering with the file, but during processing its all open. If that is a problem, you'd be better of redesigning your system so the sensitive processing is done on the server. — AviD, Jan 12 '11 at 12:00

score 1 · Answer 4 · answered Jan 04 '11 at 17:35

Apparently, there's no way to do it with the standard library without significant recoding of stuff already available in Bouncy Castle, for instance.

Here's what I came up with. On the client side, I encrypt a file plainfile as follows:

// install security provider
if (Security.getProvider(BouncyCastleProvider.PROVIDER_NAME) == null) {
    Security.addProvider(new BouncyCastleProvider());
}

// load public certificate for signing
KeyStore publicKs = KeyStore.getInstance(KeyStore.getDefaultType());
publicKs.load(new FileInputStream("/path/to/publicKeystore.ks"), null);
TrustedCertificateEntry entry = (TrustedCertificateEntry) publicKs.getEntry("public", null);
X509Certificate cert = (X509Certificate) entry.getTrustedCertificate();

// create CMS envelope data;
// check http://www.ietf.org/rfc/rfc3852.txt pages 15-16 for details
CMSEnvelopedDataGenerator envelopedDataGen = new CMSEnvelopedDataGenerator();

// specify that generated symmetric key will be encrypted by with this public key
envelopedDataGen.addKeyTransRecipient(cert);

// automatically generate the AES key, encrypt the data, and encrypt the key
CMSEnvelopedData data = envelopedDataGen.generate(new CMSProcessableFile(plainfile),
        CMSEnvelopedDataGenerator.AES256_CBC, 256, BouncyCastleProvider.PROVIDER_NAME);

byte[] encryptedData = data.getEncoded();

Then, on the server, I can decrypt this data as follows:

// load private key
KeyStore privateKs = KeyStore.getInstance(KeyStore.getDefaultType());
privateKs.load(new FileInputStream("/path/to/privateKeystore.ks"), null);
PrivateKeyEntry privateKeyEntry = (PrivateKeyEntry) privateKs.getEntry("privatekey",
        new KeyStore.PasswordProtection("password".toCharArray()));
PrivateKey privateKey = privateKeyEntry.getPrivateKey();

byte[] encryptedData = ...

// parse CMS envelope data
CMSEnvelopedDataParser envelopedDataParser = new CMSEnvelopedDataParser(new ByteArrayInputStream(encryptedData));

// expect exactly one recipient
Collection<?> recipients = envelopedDataParser.getRecipientInfos().getRecipients();
if (recipients.size() != 1)
    throw new IllegalArgumentException();

// retrieve recipient and decode it
RecipientInformation recipient = (RecipientInformation) recipients.iterator().next();
byte[] decryptedData = recipient.getContent(privateKey, BouncyCastleProvider.PROVIDER_NAME);

Please leave comments if I'm missing something important or am making mistakes here... Thanks!

A file with such encrypted contents ciperfile.enc can be decrypted with openssl: openssl cms -decrypt -in cipherfile.enc -inform DER -recip testcert.pem -inkey testkey.pem, where testcert.pem is the certificate in the public keystore in PEM format, and testkey.pem the corresponding private key. — Jean-Philippe Pellet, Jan 05 '11 at 09:30
I'm concerned that this code snippet does not seem to provide message authentication/integrity protection. If so, it is likely to be insecure against active attacks, and thus not a recommended solution. — D.W., Jan 08 '11 at 00:37
@D.W. Thanks for pointing this out. How can I fix it? Sign it after encryption? — Jean-Philippe Pellet, Jan 12 '11 at 11:30
Yes, signing after encrypting would provide authentication/integrity, if done right. However this is tricky. The best way to fix it is to rely upon some existing, well-vetted solution for file encryption, like PGP/GPG, or for channel encryption, like SSL/TLS. — D.W., Jan 19 '11 at 18:28
See this post for further explanation of why it is a bad idea to encrypt without also authenticating and how to fix the problem. It is worth emphasizing that the sample code listed above appears to be flawed and should not be used. — D.W., Feb 28 '11 at 09:19

What's the standard way to encrypt a file with a public key in Java?

4 Answers4

Linked