en-US/about_Elastic.Helper.help.txt
|
TOPIC
about_elastic.helper SHORT DESCRIPTION A PowerShell Helper Module to interface to ElasticSearch LONG DESCRIPTION Interfacing to ElasticSearch is complex. Even with the Elastic.Console module. This module solves some problems for me in publishing resources to ElasticSearch. This includes: * Index Properties * Ingest Pipelines * Enrichment Policies * Bulk Index requests This module supports both Authenticated and Unauthenticated connections to ElasticSearch. Which mode you use depends on your ElasticSearch implementation. This module is not intended to be an exhaustive implementation of ElasticSearch functionality. It was built as a way for me to be able to use Kibana, etc, to build up a working ElasticSearch configuration to meet a specific set of needs, and to be able to reproduce on an additional system. The configuration file structure defines high-level resource types, and at the resource definition level supports whatever JSON is generated by using Kibana. This module leverages the Elastic.Console PowerShell module. INDEX PROPERTIES Index properties can specify which pipeline to use for indexing, as well as an ElasticSearch index properties section. This allows you to specify, among other things, the number of replicas that should exist for that index, as well as any other, configurable index property in ElasticSearch. INGEST PIPELINES An Ingest Pipeline in ElasticSearch defines how the data is to be indexed. There are a number of factors that may be used: * Field Mappings * Date field to use as the primary index date * Field to use as Document ID to prevent duplicates * Index naming, including date-part * Enrichment Policies to apply Anything that you can specify in Kibana using the Index Definition can be exported as JSON, and saved as the definition element of the Pipeline in the configuration. ENRICHMENT POLICIES An enrichment policy allows you to add data to an index as it is being added. This usually takes the form of a lookup in another index. Enrichment policies are, therefore, usually based on an index, and specify which fields should be added to the index being eriched. Whenever the base index is updated, the Enrichment Policy needs to be refreshed in order for the system indices that implement the enrichment lookup to be updated. There are some helper functions for this. * Rebuild-EsEnrichmentIndices - will rebuild all enrichment indices based on the provided configuration * Update-EsEnrichmentIndicesFromInxed - will rebuild enrichment indices based on a specific index in the provided configuration BULK INDEX REQUESTS Allows for the bulk insert of records to an ElasicSearch index. Currently requires the use of Pipelines. EXAMPLES EXAMPLE 1 - DEPLOY CONFIGURATION TO ELASTICSEARCH Take the defined configuration: index; pipeline; enrichment; assess any dependencies, and deploy the configuration for which dependencies are met to ElasticSearch. Import-Module Elastic.Helper # Force TLS 1.2 [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 $ConfigName = 'elasticproject' $CliXmlCreds = '/path/to/clixml.xml' # Load configuration and credentials $EsConfig = Get-EsHelperConfig -ConfigName $ConfigName $EsCred = Import-CliXml -Path $CliXmlCreds Set-ElasticSearchVersion '7.10' # Deploy non-dependent elements of defined configuration Deploy-EsConfig -EsConfig $EsConfig -EsCreds $EsCred EXAMPLE 2 - BULK INDEX DATA USING SPECIFIED PIPELINE Get some SQL data from a database and send it to ElasticSearch via a Bulk Index Request. Import-Module Elastic.Helper Import-Module SqlServer # Force TLS 1.2 [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12 $ConfigName = 'elasticproject' $CliXmlEsCreds = '/path/to/clixmles.xml' $CliXmlSqlCreds = '/path/to/clixmlsql.xml' $QueryFile = '/path/to/sqlscript.sql' $ServerInstance = 'SQLServerName' # Load configuration and credentials $EsConfig = Get-EsHelperConfig -ConfigName $ConfigName $EsCred = Import-CliXml -Path $CliXmlEsCreds Set-ElasticSearchVersion '7.10' # Get SQL results $Data = Invoke-SqlCmd -Credential $CliXmlSqlCreds -ServerInstance $ServerInstance -Database 'Database' -InputFile $QueryFile # Strip off extraneous elements that cause issues indexing to Elastic $DataBare = $Data | Select-Object * -ExcludeProperty ItemArray, Table, RowError, RowState, HasErrors # Index the data via Bulk Index Request, using the pipeline specified in the $EsConfig definition of the index $output = Invoke-EsBulkIndexRequest -EsConfig $EsConfig -IndexName 'myindex' -InputObject $DataBare -EsCreds $LrAieConfig.EsCred -Debug # Error Handling if ($output.errors) { $msg = "$(Get-Date) - Warning: Errors encountered.`n {0} Statistics Records synchronised in {1}ms" -f $output.items.Count,$Output.took # Extract the items that had errors so you can do something with them $output.items.index | Where-Object {$_.status -ge 400} } else { $msg = "$(Get-Date) - {0} Statistics Records synchronised in {1}ms" -f $output.items.Count,$Output.took } Write-Output $msg NOTE This module is a work in progress, and is not complete. There are likely to be bugs and insufficient error checking/handling. SEE ALSO There will likely be modules releases that leverage this. Keep an eye on my GitHub page: jberkers42 on GitHub |