CryptoNG UDF - Cryptography API: Next Gen

[TheXman]

  On 7/10/2020 at 7:45 AM, RTFC said:

A praise-sandwich for CryptoNG:

Top bread slice: Great work; thorough, methodical, consistent, clean code, and well-documented; an excellent contribution all-round.

Expand  

@RTFC  Thanks so much! 

  On 7/10/2020 at 7:45 AM, RTFC said:

At the moment, the library does not handle UTF-8 strings correctly

Expand  

I will take a look into this.  I think CryptoNG handles UTF8 strings, but from what you have shown it may not handle strings with multi byte characters correctly.  If that is true, which it appears that it is, I will see what I can do to add the ability to handle such strings. 

In typical American fashion and since I don't ever use multi-byte character sets, for the most part, they don't exist.  It's sort of like how some of us assume that everyone speaks English.

  On 7/10/2020 at 7:45 AM, RTFC said:

Furthermore, you cannot properly test this using your current example suite, as you use Consolewrite for examining output, which would require a registry hack each time to select the appropriate code page(s) to render non-English UTF8 characters correctly.

Expand  

Yes, this is true.  By default, the Scite4AutoIt editor's message area does not correctly display multi-byte characters.  As I mentioned above, I'm sure I used ConsoleWrites in my examples because I didn't take into account the possibility of strings with multi-byte characters in them.  It was definitely short-sighted on my part.  I will see if I can come up with a better set of examples that will work universally with either single or multi-byte string output.

 

  On 7/10/2020 at 7:45 AM, RTFC said:

BTW, your recent "upgrade" of filling struct members using dot notation causes my scripts to error out (on every dot-based struct access) with:

"<path>\CryptoNG.v1.5.5.au3" (2743) : ==> Variable must be of type "Object".:
$tInputBuffer.data = Binary($vData)

Expand  

Those look like Au3Check errors and I do not get them when I run the UDF or the example file through Au3Check.  I'm not sure why you are getting them. 

I did see that I forgot to run the latest version through Au3Check with the strictest settings which allowed some "declared, but not used in func" warnings to get through.  I will definitely get those corrected in the next version.

As for using dot notation for the dll struct gets & puts, I have read and seen small tests that show it is a little faster but not by much.  In this particular case, I don't think speed is a concern because the gets & puts are not occurring in a loop or being done multiple times during encryption or decryption.  My primary reason for making the change was that it looked cleaner and required far less code to get the same result. 

  On 7/10/2020 at 7:45 AM, RTFC said:

Bottom bread slice: As you know, I've gratefully incorporated your library into my own CodeCrypter environment, and it's proving a real asset, so many, many thanks for your continuing efforts, and I hope you'll keep maintaining it.

Expand  

I'm glad that you have found the UDF useful.  I definitely plan to maintain it for the foreseeable future. 

 

I will start looking into these issue today.  Hopefully I can get them resolved quickly. 

I won't be truly satisfied with this "Praise Sandwich" until all of the fillings are as good as the slices of bread. 

 

Edited July 10, 2020 by TheXman

[TheXman]

@RTFC

I have made all of the changes to needed to handle multi-byte characters used as input to the CryptoNG functions.  I just have to do a bit of regression testing to make sure that nothing was broken by all of the modifications.  Hopefully, I will have a new version available by the end of the day tomorrow (Central Standard Time). 

[TheXman]

What's New

v1.6.0 (2020-07-10)

• Added the ability to handle data that contains multi-byte character sets. (Reported by RTFC)
• Removed all AU3CHECK warnings.
• Added a new example to show the encryption/decryption of strings with multi-byte characters: aes_cbc_encrypt_decrypt_multibyte_example()
• Added multi-byte characters to the example Word .docx so that the example script that encrypts/decrypts a file shows that it can handle multi-byte characters.
• The example scripts used to write their output to the console.  The Scite4AutoIt's editor does not display multi-byte characters in the message area.  So the example scripts now sends messages to notepad, which does handle multi-byte characters.  (Best to use a monospaced font in Notepad, like Consolas, so that the message formatting displays correctly)
• Removed a few examples whose functionality was duplicated in other example scripts.

Edited July 12, 2020 by TheXman

[RTFC]

  On 7/10/2020 at 8:11 PM, TheXman said:

new version available

Expand  

Wow, you really are as fast as your avatar picture suggests; velocius quam asparagi conquantur. Many, many thanks.

EDIT: So I found the bug triggering the error (which wasn't due to AU3check): your code never checks for empty input strings in your encryption UDFs (and I initialise my environment by calling with an empty argument), so DllStructCreate tries to allocate a buffer of size 0, which fails, but you don't check for errors after that call either. Example:

Func aes_cbc_encrypt_decrypt_example()

    Const $ALG_ID       = $CNG_BCRYPT_AES_ALGORITHM, _
          $MESSAGE      = "", _
          $SECRET       = "Secret Password!", _
          $SECRET_SALT  = "Secret Salt"

I would just add a size>0 check on the parsed string at the highest calling level. Sorry for not reporting this earlier.

Edited July 11, 2020 by RTFC
persistent bug

[TheXman]

  On 7/11/2020 at 7:54 AM, RTFC said:

So I found the bug triggering the error (which wasn't due to AU3check): your code never checks for empty input strings in your encryption UDFs (and I initialise my environment by calling with an empty argument), so DllStructCreate tries to allocate a buffer of size 0, which fails, but you don't check for errors after that call either.

Expand  

I always wondered what, if any, differences there were between using the DllStruct Get & Set functions and their dot-notation alternatives.  Thanks to you, now I know at least one difference.     The object-based, dot-notation form does not like having empty values being set.  The regular functions handle it without any issues.  I'll have to tuck that new bit of knowledge away for future reference.  Thanks for making me aware of it.

With the knowledge above, I have reverted all of the dot-notation references back to their DllStruct* functions.  I think there are too many "gotchas" that could pop up using the dot-notation.  I will be publishing v1.6.1 now.

As you may or may not be are aware, trying to encrypt an empty string has always and will continue to generate an API error.  The error says something like "...an invalid parameter has been passed".  So during your environment initialization, you will have to work around that, if you haven't already.

Update:

I went back and took a look at what you said and the root cause of the issue.  As you said, because an empty value had been passed, when I tried to set the DllStruct's buffer to zero length (byte[0]), of course it failed.  So it was really was an oversight on my part that I didn't check for empty values being passed in order to prevent that sort of error.  I should've known better.  So it really isn't an issue with object-based dot-notation. 

I've reverted everything back to dot-notation and added the additional logic validation logic to make sure that empty values are handled appropriately.  Thanks again for pointing out the issue.

Edited July 13, 2020 by TheXman
Corrected my previous statement

[TheXman]

What's New

• v1.6.1 (2020-07-11)
  • Reverted all dll struct gets & sets from dot-notation back to DllStructGetData & DllStructSetData.  Using dot-notation caused object initialization errors when value was set to an empty string. (Reported by @RTFC)

[RTFC]

  On 7/11/2020 at 1:09 PM, TheXman said:

Reverted all dll struct gets & sets

Expand  

That's funny, I just spent the last hour doing exactly the same thing (again). Yeah, I always avoid dot notation if possible (hate blackboxes).

  On 7/11/2020 at 1:05 PM, TheXman said:

So during your environment initialization, you will have to work around that

Expand  

Thanks (was aware of that); my initialisation just needs your function to be called for it to be recognised as "active" in the context of CodeScanner; otherwise it gets pruned as redundant when CodeCrypter is reconstructing the script from scratch. From now on, I'll be parsing a non-empty dummy string, just to be sure.

[TheXman]

What's New

• v1.6.2 (2020-07-12)
  • Added additional function parameter validation to prevent the issue, reported by RTFC, where passing empty strings to some functions was causing DllStructCreate failures.
  • Reverted all DllStructGetData & DllStructSetData functions back to object-based dot-notation.

[CarlD]

Hi, I am trying to use func  _CryptoNG_EncryptData() in a script, but it returns error 2 ("Unable to get handle to algorithm provider"). The output of func _CryptoNB_EnumRegisteredProviders is: 

  Quote

Microsoft Key Protection Provider
Microsoft Passport Key Storage Provider
Microsoft Platform Crypto Provider
Microsoft Primitive Provider
Microsoft Smart Card Key Storage Provider
Microsoft Software Key Storage Provider
Microsoft SSL Protocol Provider
Windows Client Key Protection Provider

Expand  

My code looks like this:

$vOut = _CryptoNG_EncryptData($vIn, $sKey, $CNG_BCRYPT_AES_ALGORITHM, "Microsoft Primitive Provider")
If @error Then Exit MsgBox(0, "", @error)

I'm using a 256-bit key. I get the same error 2 with "Microsoft Key Protection Provider". I have not tried any of the other listed providers.

My system specs:

  Quote

AutoIt Version: V3.3.14.5 [X64]
Windows Version: WIN_10 [X64] [Version 10.0.18363.1016]
Language: 0409

Expand  

Any help would be much appreciated. (I hope this is the appropriate forum to post this question.)

 

Edited August 29, 2020 by CarlD

[TheXman]

If I may take a step back and ask a question, if you want to use AES, why aren't you using the AES-specific function (_CryptoNG_AES_CBC_EncryptData) and using its example script in the CryptoNG help file as a starting point?

 

  On 8/29/2020 at 9:31 PM, CarlD said:

$vOut = _CryptoNG_EncryptData($vIn, $sKey, $CNG_BCRYPT_AES_ALGORITHM, "Microsoft Primitive Provider")

Expand  

To answer your initial question, I'm not sure how you came up with that parameter list for that function.  The CryptoNG help file shows the following for the function:

_CryptoNG_EncryptData($sAlgorithmId, $sText, $vEncryptionKey, $sProvider = Default)

 

But as I mention before, I would use algorithm-specific functions over the more generic Encryption/Decryption functions, if they exist.  Which in this particular case, one does exist.  If you are determined to use that function, look at the example for _CryptoNG_EncryptData  in the CryptoNG help file and use that as a starting point.  It is an example of AES encryption but uses a 192-bit encryption key.  It can be easily modified to use a 256-bit encryption key.

Edited August 29, 2020 by TheXman

[CarlD]

  On 8/29/2020 at 9:46 PM, TheXman said:

To answer your initial question, I'm not sure how you came up with that parameter list for that function. 

Expand  

A stupid, inattentive error on my part. Sorry for wasting your time.

In the meantime, I  have been getting along fine with func _CryptoNG_AES_CBC_EncryptData and its Decrypt counterpart -- using the correct syntax. 

Thanks very much for your quick reply.

[TheXman]

You're welcome!  I've been there before.  I'm glad that you've found the UDF useful. 

Edited August 31, 2020 by TheXman

[CarlD]

  On 8/29/2020 at 10:05 PM, TheXman said:

I'm glad that you've found the UDF useful. 

Expand  

And I'm glad I came across it. After some further testing I plan to incorporate it into my DIY password manager. It'll give it a new lease on life once the deprecated encryption functions are taken out of service. I'll definitely be following this UDF for further developments! 

[CarlD]

  On 8/29/2020 at 10:05 PM, TheXman said:

 

@CarlD Do you really think my previous response warranted a "Sad" emoticon?  😃

Expand  

I meant it as "WTF was I thinking when I posted that stupid question?". Changed to Thank you icon now. 

[mLipok]

@TheXman please take a look here:

https://www.debenu.com/kb/advanced-options-signing-pdf-files/

I'm wondering if we could use this code part in AutoIt and of course change to use windows store instead PFX file and then use smartcard

How you think could we do this with AutoIt + your UDF ?

' Load a certificate from a .pfx file
        ' This could be changed to use a certificate from the store
 
        cert = New X509Certificate2(workFolder + "qpl_test.pfx", "testing")
        AddLog("Loaded certificate: " + cert.SubjectName.Name)
 
        ' Estimate how much space we need for the signature by first signing some
        ' random data
 
        RandomData(1) = 123
        TestSig = SignData(RandomData, cert)

Private Function SignData(inputData As Byte(), cert As X509Certificate2) As Byte()
 
        Dim content As ContentInfo
        Dim SignedCms As SignedCms
        Dim CmsSigner As CmsSigner
        Dim sha As SHA1
        Dim sha1Result As Byte()
 
        ' Create an SHA-1 hash of the file data
        sha = New SHA1CryptoServiceProvider()
        sha1Result = sha.ComputeHash(inputData)
 
        ' Sign the hash using the certificate
        ' This could be changed to use a hardware device (eg. smartcard)
        content = New ContentInfo(sha1Result)
        SignedCms = New SignedCms(content)
        CmsSigner = New CmsSigner(cert)
        SignedCms.ComputeSignature(CmsSigner)
 
        Return SignedCms.Encode()
 
    End Function

 

Edited October 20, 2020 by mLipok

[TheXman]

  On 10/20/2020 at 7:13 PM, mLipok said:

I'm wondering if we could use this code part in AutoIt and of course change to use windows store instead PFX file and then use smartcard

How you think could we do this with AutoIt + your UDF ?

Expand  

@mLipok

I will take a look at the information on the site you provided, the MS CNG functions that might be applicable, and any other relevant information that I can find about PDF signing and see what's possible.  If I understand you correctly, at least initially, you would prefer to use a public/private key from the local store rather than supplying a public/private key pfx file to a signing function?

If you had a choice, which would you prefer first, the ability to sign a PDF file or the ability to verify the signature of a PDF file?

 

Edit:

At first glance, MS' CNG APIs have the ability to sign & verify data.  Looking at the sample from the link, at least half of the functionality already exists in the UDF.  Since I haven't ever used the signing & verification functionality of the CNG APIs, it would take a bit of time to understand them.  Once I understand them, implementing them in the UDF shouldn't be too difficult.  However, it appears that it is talking about signing "data", any data.  If there is something special dealing with PDF's or that you are looking for, I would need to understand that too.

Edited October 20, 2020 by TheXman

[mLipok]

  On 10/20/2020 at 7:31 PM, TheXman said:

you would prefer to use a public/private key from the local store rather than supplying a public/private key pfx file to a signing function?

Expand  

I want ot use Certificate from windows storage

  Quote

  Quote

            // This could be changed to use a certificate from the store
            X509Certificate2 cert = new X509Certificate2(workFolder + "qpl_test.pfx", "testing");

Expand  

 

Expand  

which is related to private key on smart card

  Quote

        ' Sign the hash using the certificate
        ' This could be changed to use a hardware device (eg. smartcard)
        content = New ContentInfo(sha1Result)
        SignedCms = New SignedCms(content)
        CmsSigner = New CmsSigner(cert)
        SignedCms.ComputeSignature(CmsSigner)
 

Expand  

In this way inputData shoube be signed using Cert+Smartcard

  Quote

    Private Function SignData(inputData As Byte(), cert As X509Certificate2) As Byte()
......
        ' Sign the hash using the certificate
        ' This could be changed to use a hardware device (eg. smartcard)
......
        Return SignedCms.Encode()
 
    End Function

Expand  

This would be realy helpfull not only for PDF/PADES signing but also for any kind of data signing proces (like XADES - XML file signed with QualifiedCerts + SmartCard + PIN).

 

[TheXman]

@mLipok

I've got a bit of free time at the moment so I'll look into adding general use data signing and verification routines. 

I just finished adding a new function to enumerate registered key storage providers.  It is just like the current functions that enumerate registered cryptography providers & algorithms.  Next will be adding a function to enumerate registered signing algorithms and then I will do the signing and verification functions.

[TheXman]

I've had the time to do a little more research into what all is involved in file signing using CNG.  A good summary of the information can be found in the following links:

• CNG: Key Storage & Retrieval
• CNG: Key Import & Export
• CNG: Key Storage Functions

The older Microsoft CryptoAPI and the newer Microsoft CNG API store their keys in different places and using different data structures.  If I were to implement CNG file signing correctly, I would need to be able to handle both the older and newer key store formats (and locations).  To be honest, since I don't have a need for it, file signing just doesn't interest me enough to spend the time to implement it the way it needs to be done.  If you or anyone else wants to write the missing pieces to implement file signing, I would be happy to review, standardize, optimize, and add it to the CryptoNG UDF with the appropriate acknowledgements.  I'm not saying that I won't do it.  However, for me to spend the time necessary to do it right, it would have to be of interest to me like if I somehow developed a need to sign files on an ongoing basis, it was a widely requested feature in the forum, I'm getting paid to do it, or I suddenly felt inspired to a bunch of coding and had no other coding ideas on my list.   Of course I could always slap some single use case functionality into the UDF just to say that it was implemented, but that is not my style.  If I'm going to do it, I'm going to do it right and to the best of my abilities.

 

For anyone interested in adding file signing functionality to the UDF, at a high level, below are the steps required to do it.  I have identified the functionality that already exists in the CryptoNG UDF.  Some of the required Ncrypt functions or almost a complete copy & paste of existing Bcrypt functions like NCryptOpenStorageProvider, which is almost exactly the same as BCryptOpenAlgorithmProvider.

1. Hash a file/data (already exists)
2. Create or select a public/private key pair (partially exist)
   1. The ability to create ephemeral RSA key pairs already exists (Bcrypt).
   2. The ability to create persisted key/pairs (key pairs that are save to a key store) does not currently exist (Ncrypt).
   3. Note: The main difference between Bcrypt & NCrypt functions as it relate to key storage, is that  Bcrypt functions work with ephemeral keys and Ncrypt works with persisted keys.
3. Sign the hash with the private key, to create a signature. (does not exist)
4. Provide the signature to a user in a usable format. (does not exist)
5. Be able to verify a signature. (does not exist)

Edited October 21, 2020 by TheXman

[TheXman]

Posted v1.7.0 to the downloads area. 

v1.7.0 has mostly minor maintenance fixes to some function headers and some additional debug logging.  The only new functionality is a function that enumerates key storage providers and an internal function that supports the new function.  This new functionality was added in preparation for possibly adding functions that will do data signing and data signature verification.