Summary of Changes:

- moved es install/start/stop logic into a seperate script
  - `grunt test` now runs the integration tests once for each version of ES we support
  - grunt can now install and run elasticearch (using grunt-run, pure js solution coming later)
  - included seperate es.sh script specifically for starting or stopping elasticsearch
  - url aliases, api, yaml_suite/index.js, and yaml_tests.json, are all now duplicated for 0_90 support
  - the client now accepts an apiVersion argument (undocumented) which defaults to 'master' but can be '0.90'
  - The yaml test runner will now check the name of the ES instance it is connecting to, preventing accidental wiping of ES
This commit is contained in:
Spencer Alger
2014-01-14 23:10:12 -07:00
parent 37ce4e440c
commit 18e134d0a6
30 changed files with 23318 additions and 560 deletions

View File

@ -1079,9 +1079,9 @@ The default method is `DELETE` and the usual <<api-conventions,params and return
`masterTimeout`::
`Date, Number` -- Specify timeout for connection to master
`index`::
`String` -- The name of the index with an alias
`String, String[], Boolean` -- A comma-separated list of index names (supports wildcards); use `_all` for all indices
`name`::
`String` -- The name of the alias to be deleted
`String, String[], Boolean` -- A comma-separated list of aliases to delete (supports wildcards); use `_all` to delete all aliases for the specified indices.
[[api-indices-deletemapping]]
=== `indices.deleteMapping`
@ -1104,9 +1104,9 @@ The default method is `DELETE` and the usual <<api-conventions,params and return
`masterTimeout`::
`Date, Number` -- Specify timeout for connection to master
`index`::
`String, String[], Boolean` -- A comma-separated list of index names; use `_all` for all indices
`String, String[], Boolean` -- A comma-separated list of index names (supports wildcards); use `_all` for all indices
`type`::
`String` -- The name of the document type to delete
`String, String[], Boolean` -- A comma-separated list of document types to delete (supports wildcards); use `_all` to delete all document types in the specified indices.
[[api-indices-deletetemplate]]
=== `indices.deleteTemplate`
@ -1153,12 +1153,10 @@ The default method is `DELETE` and the usual <<api-conventions,params and return
[horizontal]
`masterTimeout`::
`Date, Number` -- Specify timeout for connection to master
`index`::
`String, String[], Boolean` -- A comma-separated list of index names to register warmer for; use `_all` or empty string to perform the operation on all indices
`name`::
`String` -- The name of the warmer (supports wildcards); leave empty to delete all warmers
`type`::
`String, String[], Boolean` -- A comma-separated list of document types to register warmer for; use `_all` or empty string to perform the operation on all types
`String, String[], Boolean` -- A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters.
`index`::
`String, String[], Boolean` -- A comma-separated list of index names to delete warmers from (supports wildcards); use `_all` to perform the operation on all indices.
[[api-indices-exists]]
=== `indices.exists`
@ -1574,7 +1572,7 @@ The default method is `PUT` and the usual <<api-conventions,params and return va
`masterTimeout`::
`Date, Number` -- Specify timeout for connection to master
`index`::
`String` -- The name of the index with an alias
`String, String[], Boolean` -- A comma-separated list of index names the alias should point to (supports wildcards); use `_all` or omit to perform the operation on all indices.
`name`::
`String` -- The name of the alias to be created or updated
@ -1609,7 +1607,7 @@ The default method is `PUT` and the usual <<api-conventions,params and return va
`[expandWildcards=open]`::
`String` -- Whether to expand wildcard expression to concrete indices that are open, closed or both.
`index`::
`String, String[], Boolean` -- A comma-separated list of index names; use `_all` to perform the operation on all indices
`String, String[], Boolean` -- A comma-separated list of index names the mapping should be added to (supports wildcards); use `_all` or omit to add the mapping on all indices.
`type`::
`String` -- The name of the document type
@ -1700,7 +1698,7 @@ The default method is `PUT` and the usual <<api-conventions,params and return va
`[expandWildcards=open]`::
`String` -- Whether to expand wildcard expression to concrete indices that are open, closed or both, in the search request to warm.
`index`::
`String, String[], Boolean` -- A comma-separated list of index names to register the warmer for; use `_all` or empty string to perform the operation on all indices
`String, String[], Boolean` -- A comma-separated list of index names to register the warmer for; use `_all` or omit to perform the operation on all indices
`name`::
`String` -- The name of the warmer
`type`::

2572
docs/api_methods_.asciidoc Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@ -1,12 +1,16 @@
var unitTests = ['test/unit/test_*.js'];
var integrationTests = ['test/integration/yaml_suite/index.js'];
module.exports = {
unit: {
src: unitTests
},
integration: {
src: integrationTests
integration_master: {
src: 'test/integration/yaml_suite/index.js'
},
integration_0_90: {
src: 'test/integration/yaml_suite/index_0_90.js'
},
// run the unit tests, and update coverage.html

View File

@ -1,10 +1,20 @@
var esOpts = [
'-D es.network.host=localhost',
'-D es.cluster.name=elasticsearch_js_test_runners',
'-D es.node.name=elasticsearch_js_test_runner',
'-D es.gateway.type=none',
'-D es.index.store.type=memory',
'-D es.discovery.zen.ping.multicast.enabled=false',
'-D es.discovery.zen.ping_timeout=1',
'-D es.logger.level=ERROR'
].join(' ');
module.exports = {
generate: {
exec: 'node ./scripts/generate/index.js',
options: {
passArgs: [
'verbose',
'es_branch'
'verbose'
]
}
},
@ -12,8 +22,7 @@ module.exports = {
exec: 'node ./scripts/generate/index.js --no-api',
options: {
passArgs: [
'verbose',
'es_branch'
'verbose'
]
}
},
@ -23,5 +32,31 @@ module.exports = {
wait: false,
ready: /listening/
}
},
install_es_master: {
exec: './scripts/es.sh install master',
},
es_master: {
exec: './.snapshots/master_nightly/bin/elasticsearch ' + esOpts,
options: {
wait: false,
quiet: true,
onClose: function () {
},
onReady: function () {
}
}
},
install_es_0_90: {
exec: './scripts/es.sh install 0.90',
},
es_0_90: {
exec: './.snapshots/0.90_nightly/bin/elasticsearch -f ' + esOpts,
options: {
wait: false,
quiet: true
}
}
};

View File

@ -7,9 +7,18 @@ module.exports = function (grunt) {
grunt.registerTask('test', [
'jshint',
'mochacov:unit',
'run:generate',
'mochacov:integration',
'mochacov:unit',
'run:install_es_master',
'run:es_master',
'mochacov:integration_master',
'stop:es_master',
'run:install_es_0_90',
'run:es_0_90',
'mochacov:integration_0_90',
'stop:es_0_90'
]);
grunt.registerTask('unit_test', [

View File

@ -56,7 +56,8 @@
"when": "~2.6.0",
"lodash": "~2.3.0",
"agentkeepalive": "~0.1",
"chalk": "~0.3.0"
"chalk": "~0.3.0",
"event-stream": "~3.1.0"
},
"repository": {
"type": "git",

126
scripts/_utils.sh Normal file
View File

@ -0,0 +1,126 @@
#####
# Start or stop a group for travis
#####
function group {
if [ -n "$TRAVIS" ]; then
echo -e "travis_fold:$1"
fi
}
#####
# Do, log, and check a call
#####
function call {
local DO="$*"
echo "\$ ${DO}"
echo $DO | bash
local RESULT=$?
if [ "$RESULT" -gt "0" ]; then
echo "non-zero exit code: $RESULT"
exit $RESULT
fi
}
#####
# Download a version of ES and get it running
# @arg ES_BRANCH - The branch to run off of
# @arg ES_RELEASE - The specific release to run, overrides ES_BRANCH
#####
function manage_es {
local DO=$1
local ES_BRANCH=$2
local ES_RELEASE=$3
local ROOT="$PWD"
local ES_SUBMODULE="$ROOT/src/elasticsearch"
local SNAPSHOTS="$ROOT/.snapshots"
local PIDS="$ROOT/.snapshots/pids"
group "start:$DO es"
if [ ! -d "$PIDS" ]; then
call mkdir -p $PIDS
fi
if [ -n "$ES_RELEASE" ]; then
local ES_VERSION="v${ES_RELEASE}"
local ES_URL="https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ES_RELEASE}.zip"
local ES_DIR="${SNAPSHOTS}/${ES_VERSION}"
else
local ES_VERSION="${ES_BRANCH}_nightly"
local ES_URL="http://s3-us-west-2.amazonaws.com/build.elasticsearch.org/origin/$ES_BRANCH/nightly/JDK6/elasticsearch-latest-SNAPSHOT.zip"
local DATE=`date +%Y_%m_%d`
local ES_DIR="${SNAPSHOTS}/${ES_VERSION}_${DATE}"
fi
local ES_BIN="$ES_DIR/bin/elasticsearch"
local PIDFILE="$ROOT/.snapshots/pids/$ES_VERSION"
case "$DO" in
install)
if [ ! -d "$ES_DIR" ]; then
echo "Downloading Elasticsearch $ES_VERSION"
call rm -rf ${SNAPSHOTS}/${ES_VERSION}*
call curl -#O $ES_URL
unzip -q elasticsearch-*.zip
rm elasticsearch-*.zip
mv elasticsearch-*/ $ES_DIR
if [ -z "$ES_RELEASE" ]; then
ln -sf $ES_DIR "${SNAPSHOTS}/${ES_VERSION}"
fi
fi
;;
start)
# ensure that only one version is running at a time so that we can precisely kill them
if [ -f $PIDFILE ]; then
local PID=`cat $PIDFILE`
kill -0 $PID
local RUNNING=$?
if [ $RUNNING -eq 0 ]; then
echo "Already running $ES_VERSION"
exit 1
else
echo "PID file was left behind by ES"
call rm $PIDFILE
fi
fi
./scripts/es.sh install $ES_BRANCH $ES_RELEASE
if [ ! -x "$ES_BIN" ]; then
echo "Unable to find elasticsearch executable"
exit 1
fi
local ES_OPTS="-p $PIDFILE -Des.network.host=localhost -Des.discovery.zen.ping.multicast.enabled=false -Des.discovery.zen.ping_timeout=1"
if [ "$ES_BRANCH" = "0.90" ]; then
echo "Starting Elasticsearch $ES_VERSION"
call $ES_BIN $ES_OPTS
else
echo "Starting Elasticsearch $ES_VERSION as a deamon"
call $ES_BIN -d $ES_OPTS
fi
;;
stop)
if [ -e $PIDFILE ]; then
local PID=`cat $PIDFILE`
kill -0 $PID
local RUNNING=$?
if [ $RUNNING -eq 0 ]; then
kill $PID
echo "Elasticsearch $ES_VERSION stopped"
exit 0
fi
rm $PIDFILE
fi
echo "Elasticsearch $ES_VERSION is not running."
exit 1
;;
esac
group "end:$DO es"
}

View File

@ -15,28 +15,7 @@
#
###########
#####
# Start or stop a group for travis
#####
function group {
if [ -n "$TRAVIS" ]; then
echo -e "travis_fold:$1"
fi
}
#####
# Do, log, and check a call
#####
function call {
local DO="$*"
echo "\$ ${DO}"
echo $DO | bash
local RESULT=$?
if [ "$RESULT" -gt "0" ]; then
echo "non-zero exit code: $RESULT"
exit $RESULT
fi
}
source _utils.sh
#####
# call grunt, but make sure it's installed first
@ -54,72 +33,6 @@ function grunt_ {
call grunt $DO
}
#####
# Download a version of ES and get it running
# @arg ES_BRANCH - The branch to run off of
# @arg ES_RELEASE - The specific release to run, overrides ES_BRANCH
#####
function get_es {
group "start:setup_es"
local ES_BRANCH=$1
local ES_RELEASE=$2
local ROOT="$PWD"
local ES_SUBMODULE="$ROOT/src/elasticsearch"
local SNAPSHOTS="$ROOT/.snapshots"
if [ ! -d "$SNAPSHOTS" ]; then
mkdir -p $SNAPSHOTS
fi
if [ -n "$ES_RELEASE" ]; then
local ES_VERSION="v${ES_RELEASE}"
local ES_URL="https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ES_RELEASE}.zip"
local ES_DIR="${SNAPSHOTS}/${ES_VERSION}"
else
local ES_VERSION="${ES_BRANCH}_nightly"
local ES_URL="http://s3-us-west-2.amazonaws.com/build.elasticsearch.org/origin/$ES_BRANCH/nightly/JDK6/elasticsearch-latest-SNAPSHOT.zip"
local DATE=`date +%Y_%m_%d`
local ES_DIR="${SNAPSHOTS}/${ES_VERSION}_${DATE}"
if [ ! -d $ES_DIR ]; then
call rm -rf ${SNAPSHOTS}/${ES_VERSION}*
fi
fi
local ES_BIN="$ES_DIR/bin/elasticsearch"
call cd $SNAPSHOTS
if [ ! -d "$ES_DIR" ]; then
echo "Downloading Elasticsearch $ES_VERSION"
call curl -#O $ES_URL
unzip -q elasticsearch-*.zip
rm elasticsearch-*.zip
mv elasticsearch-*/ $ES_DIR
fi
call cd $ROOT
if [ ! -x "$ES_BIN" ]; then
echo "Unable to find elasticsearch executable"
exit 1
fi
if [ "$ES_BRANCH" = "0.90" ]; then
echo "Starting Elasticsearch $ES_VERSION"
call $ES_BIN \
-Des.network.host=localhost \
-Des.discovery.zen.ping.multicast.enabled=false \
-Des.discovery.zen.ping_timeout=1
else
echo "Starting Elasticsearch $ES_VERSION as a deamon"
call $ES_BIN -d \
-Des.network.host=localhost \
-Des.discovery.zen.ping.multicast.enabled=false \
-Des.discovery.zen.ping_timeout=1
fi
group "end:setup_es"
}
if [ -n "$ES_BRANCH" ]; then
TESTING_BRANCH=$ES_BRANCH
@ -135,15 +48,14 @@ fi
if [[ "$NODE_INTEGRATION" != "0" ]]; then
if [[ -n "$ES_BRANCH" ]] && [[ "$USER" != "jenkins" ]]; then
killall java 2>/dev/null
get_es $ES_BRANCH $ES_RELEASE
manage_es start $ES_BRANCH $ES_RELEASE
fi
call node scripts/generate --no-api --es_branch=\"$TESTING_BRANCH\"
grunt_ mochacov:integration
if [[ -n "$ES_BRANCH" ]] && [[ "$USER" != "jenkins" ]]; then
killall java 2>/dev/null
manage_es stop $ES_BRANCH $ES_RELEASE
fi
fi

19
scripts/es.sh Executable file
View File

@ -0,0 +1,19 @@
if [ -z "$2" ]; then
echo "Usage:
From the root of the elasticsearch-js project call:
Start nightly:
./scripts/es.sh start master
Stop 0.90 branch:
./scripts/es.sh stop 0.90
Start relase version 0.90.7:
./scripts/es.sh start 0.90 0.90.7
"
exit 1
fi
source scripts/_utils.sh
manage_es $*

View File

@ -0,0 +1,95 @@
module.exports = {
'cluster.nodeHotThreads': [
'/_cluster/nodes/hotthreads',
'/_cluster/nodes/hot_threads',
'/_nodes/hot_threads',
'/_cluster/nodes/{node_id}/hotthreads',
'/_cluster/nodes/{node_id}/hot_threads',
'/_nodes/{node_id}/hot_threads'
],
'cluster.nodeInfo': [
'/_cluster/nodes',
'/_nodes/settings',
'/_nodes/os',
'/_nodes/process',
'/_nodes/jvm',
'/_nodes/thread_pool',
'/_nodes/network',
'/_nodes/transport',
'/_nodes/http',
'/_nodes/plugin',
'/_cluster/nodes/{node_id}',
'/_nodes/{node_id}/settings',
'/_nodes/{node_id}/os',
'/_nodes/{node_id}/process',
'/_nodes/{node_id}/jvm',
'/_nodes/{node_id}/thread_pool',
'/_nodes/{node_id}/network',
'/_nodes/{node_id}/transport',
'/_nodes/{node_id}/http',
'/_nodes/{node_id}/plugin'
],
'cluster.nodeShutdown': [
'/_cluster/nodes/_shutdown'
],
'cluster.nodeStats': [
'/_cluster/nodes/stats',
'/_nodes/stats/{metric_family}',
'/_nodes/stats/indices/{metric}/{fields}',
'/_cluster/nodes/{node_id}/stats',
'/_nodes/{node_id}/stats/{metric_family}',
'/_nodes/{node_id}/stats/indices/{metric}/{fields}'
],
'get': [
'/{index}/{type}/{id}/_source'
],
'indices.deleteMapping': [
'/{index}/{type}',
'/{index}/_mapping/{type}',
'/{index}/{type}/_mappings',
'/{index}/_mappings/{type}'
],
'indices.putWarmer': [
// '/_warmer/{name}',
// '/{index}/_warmer/{name}',
// '/{index}/{type}/_warmer/{name}',
'/_warmers/{name}',
'/{index}/_warmers/{name}',
'/{index}/{type}/_warmers/{name}'
],
'indices.deleteWarmer': [
// '/{index}/_warmer/{name}',
'/{index}/_warmer',
'/{index}/_warmers',
'/{index}/_warmers/{name}'
],
'indices.deleteAlias': [
// '/{index}/_alias/{name}',
'/{index}/_aliases/{name}'
],
'indices.putAlias': [
// '/{index}/_alias/{name}',
// '/_alias/{name}',
'/{index}/_aliases/{name}',
'/_aliases/{name}'
],
'indices.putMapping': [
// '/{index}/_mapping/{type}',
// '/_mapping/{type}',
'/{index}/{type}/_mapping',
'/{index}/{type}/_mappings',
'/{index}/_mappings/{type}',
'/_mappings/{type}'
],
'indices.stats': [
'_stats/{metric_family}',
'/_stats/indexing',
'/_stats/indexing/{indexing_types}',
'/_stats/search/{search_groups}',
'/_stats/fielddata/{fields}',
'/{index}/_stats/{metric_family}',
'/{index}/_stats/indexing',
'/{index}/_stats/search/{search_groups}',
'/{index}/_stats/fielddata/{fields}'
]
};

View File

@ -0,0 +1,63 @@
module.exports = {
'cluster.nodeHotThreads': [
'/_cluster/nodes/hotthreads',
'/_cluster/nodes/hot_threads',
'/_nodes/hot_threads',
'/_cluster/nodes/{node_id}/hotthreads',
'/_cluster/nodes/{node_id}/hot_threads',
'/_nodes/{node_id}/hot_threads'
],
'cluster.nodeInfo': [
'/_cluster/nodes',
'/_nodes/settings',
'/_nodes/os',
'/_nodes/process',
'/_nodes/jvm',
'/_nodes/thread_pool',
'/_nodes/network',
'/_nodes/transport',
'/_nodes/http',
'/_nodes/plugin',
'/_cluster/nodes/{node_id}',
'/_nodes/{node_id}/settings',
'/_nodes/{node_id}/os',
'/_nodes/{node_id}/process',
'/_nodes/{node_id}/jvm',
'/_nodes/{node_id}/thread_pool',
'/_nodes/{node_id}/network',
'/_nodes/{node_id}/transport',
'/_nodes/{node_id}/http',
'/_nodes/{node_id}/plugin'
],
'cluster.nodeShutdown': [
'/_cluster/nodes/_shutdown'
],
'cluster.nodeStats': [
'/_cluster/nodes/stats',
'/_nodes/stats/{metric_family}',
'/_nodes/stats/indices/{metric}/{fields}',
'/_cluster/nodes/{node_id}/stats',
'/_nodes/{node_id}/stats/{metric_family}',
'/_nodes/{node_id}/stats/indices/{metric}/{fields}'
],
'get': [
'/{index}/{type}/{id}/_source'
],
'indices.deleteMapping': [
'/{index}/{type}/_mapping'
],
'indices.stats': [
'_stats/{metric_family}',
'/_stats/indexing',
'/_stats/indexing/{indexing_types}',
'/_stats/search/{search_groups}',
'/_stats/fielddata/{fields}',
'/{index}/_stats/{metric_family}',
'/{index}/_stats/indexing',
'/{index}/_stats/search/{search_groups}',
'/{index}/_stats/fielddata/{fields}'
],
'search': [
'/_search'
]
};

View File

@ -1,5 +1,7 @@
var cp = require('child_process');
var async = require('async');
var estream = require('event-stream');
var chalk = require('chalk');
var argv = require('optimist')
.options({
force: {
@ -39,56 +41,72 @@ if (!argv.force && process.env.FORCE || process.env.FORCE_GEN) {
argv.force = argv.f = process.env.FORCE || process.env.FORCE_GEN;
}
var branch = argv.es_branch;
// branch can be prefixed with = or suffixed with _nightly
if (branch.indexOf) {
['='].forEach(function removePrefix(pref) {
if (branch.indexOf(pref) === 0) {
branch = branch.substring(pref.length);
}
});
function spawn(cmd, args) {
console.log(chalk.white.bold('$ ' + cmd + ' ' + args.join(' ')));
['_nightly'].forEach(function removeSuffix(suf) {
if (branch.indexOf(suf) === branch.length - suf.length) {
branch = branch.substr(0, branch.length - suf.length);
}
});
var proc = cp.spawn(cmd, args, { stdio: 'pipe'});
var out = estream.split();
if (argv.verbose) {
proc.stdout.pipe(out);
} else {
proc.stdout.resume();
}
proc.stderr.pipe(out);
out
.pipe(estream.mapSync(function indent(line) {
return ' ' + line + '\n';
}))
.pipe(process.stdout);
return proc;
}
var stdio = [
'ignore',
argv.verbose ? process.stdout : 'ignore',
process.stderr
];
function generateBranch(branch, i, done) {
async.series([
function (done) {
if (i === 0) {
spawn('git', ['submodule', 'update', '--init'])
.on('exit', function (status) {
done(status ? new Error('Unable to init submodules.') : void 0);
});
return;
}
done();
},
function (done) {
spawn('git', ['submodule', 'foreach', [
'git fetch origin master', 'git reset --hard', 'git clean -fdx', 'git checkout origin/' + branch
].join(' && ')])
.on('exit', function (status) {
done(status ? new Error('Unable to checkout ' + branch) : void 0);
});
},
function (done) {
var tasks = [];
if (argv.api) {
tasks.push(
async.apply(require('./js_api'), branch)
);
}
if (argv.tests) {
tasks.push(
async.apply(require('./yaml_tests'), branch)
);
}
async.parallel(tasks, done);
}
], done);
}
async.series([
function (done) {
cp.spawn('git', ['submodule', 'update', '--init'], {
stdio: stdio
}).on('exit', function (status) {
done(status ? new Error('Unable to init submodules.') : void 0);
});
},
function (done) {
// checkout branch and clean it
cp.spawn('git', ['submodule', 'foreach', 'git fetch origin master && git checkout origin/' + branch + ' && git clean -f'], {
stdio: stdio
}).on('exit', function (status) {
done(status ? new Error('Unable to checkout ' + branch) : void 0);
});
},
function (done) {
var tasks = [];
if (argv.api) {
tasks.push(require('./js_api'));
}
if (argv.tests) {
tasks.push(require('./yaml_tests'));
}
async.parallel(tasks, done);
}
async.apply(generateBranch, 'master', 0),
async.apply(generateBranch, '0.90', 1)
], function (err) {
if (err) {
throw err;

View File

@ -1,6 +1,4 @@
var aliases; // defined at the bottom of this file.
module.exports = function (done) {
module.exports = function (branch, done) {
/**
* Read the API actions form the rest-api-spec repo.
* @type {[type]}
@ -17,6 +15,9 @@ module.exports = function (done) {
var apiSpec; // populated by parseSpecFiles
var docVars; // slightly modified clone of apiSpec for the docs
var branchSuffix = branch === 'master' ? '' : '_' + _.snakeCase(branch);
var aliases = require('./aliases' + branchSuffix);
// generate the API
async.series([
readSpecFiles,
@ -28,9 +29,11 @@ module.exports = function (done) {
], done);
function readSpecFiles(done) {
var apiDir = '../../src/elasticsearch/rest-api-spec/api/';
var apiDir = require('path').join(__dirname, '../../src/elasticsearch/rest-api-spec/api/');
files = fs.readdirSync(apiDir).map(function (filename) {
return require(apiDir + filename);
var module = require(apiDir + filename);
delete require.cache[apiDir + filename];
return module;
});
done();
}
@ -68,9 +71,10 @@ module.exports = function (done) {
}
function writeApiFile(done) {
var outputPath = require('path').join(__dirname, '../../src/lib/api.js');
console.log('writing', apiSpec.actions.length, 'api actions to', outputPath);
fs.writeFile(outputPath, templates.apiFile(apiSpec), done);
var outputPath = require('path').join(__dirname, '../../src/lib/api' + branchSuffix + '.js');
fs.writeFileSync(outputPath, templates.apiFile(apiSpec));
console.log('wrote', apiSpec.actions.length, 'api actions to', outputPath);
done();
}
function ensureDocsDir(done) {
@ -102,7 +106,7 @@ module.exports = function (done) {
function writeMethodDocs(done) {
fs.writeFile(
'../../docs/api_methods.asciidoc',
'../../docs/api_methods' + branchSuffix + '.asciidoc',
templates.apiMethods(docVars),
done
);
@ -186,7 +190,7 @@ module.exports = function (done) {
});
if (urlSignatures.length !== _.unique(urlSignatures).length) {
throw new Error('Multiple URLS with the same signature detected for ' + spec.name);
throw new Error('Multiple URLS with the same signature detected for ' + spec.name + '\n' + _.pluck(urls, 'fmt').join('\n') + '\n');
}
if (urls.length > 1) {
@ -290,68 +294,3 @@ module.exports = function (done) {
}
};
aliases = {
'cluster.nodeHotThreads': [
'/_cluster/nodes/hotthreads',
'/_cluster/nodes/hot_threads',
'/_nodes/hot_threads',
'/_cluster/nodes/{node_id}/hotthreads',
'/_cluster/nodes/{node_id}/hot_threads',
'/_nodes/{node_id}/hot_threads'
],
'cluster.nodeInfo': [
'/_cluster/nodes',
'/_nodes/settings',
'/_nodes/os',
'/_nodes/process',
'/_nodes/jvm',
'/_nodes/thread_pool',
'/_nodes/network',
'/_nodes/transport',
'/_nodes/http',
'/_nodes/plugin',
'/_cluster/nodes/{node_id}',
'/_nodes/{node_id}/settings',
'/_nodes/{node_id}/os',
'/_nodes/{node_id}/process',
'/_nodes/{node_id}/jvm',
'/_nodes/{node_id}/thread_pool',
'/_nodes/{node_id}/network',
'/_nodes/{node_id}/transport',
'/_nodes/{node_id}/http',
'/_nodes/{node_id}/plugin'
],
'cluster.nodeShutdown': [
'/_cluster/nodes/_shutdown'
],
'cluster.nodeStats': [
'/_cluster/nodes/stats',
'/_nodes/stats/{metric_family}',
'/_nodes/stats/indices/{metric}/{fields}',
'/_cluster/nodes/{node_id}/stats',
'/_nodes/{node_id}/stats/{metric_family}',
'/_nodes/{node_id}/stats/indices/{metric}/{fields}'
],
'get': [
'/{index}/{type}/{id}/_source'
],
'indices.deleteMapping': [
'/{index}/{type}/_mapping'
],
'indices.deleteWarmer': [
'/{index}/_warmer',
'/{index}/_warmers',
'/{index}/_warmers/{name}'
],
'indices.stats': [
'_stats/{metric_family}',
'/_stats/indexing',
'/_stats/indexing/{indexing_types}',
'/_stats/search/{search_groups}',
'/_stats/fielddata/{fields}',
'/{index}/_stats/{metric_family}',
'/{index}/_stats/indexing',
'/{index}/_stats/search/{search_groups}',
'/{index}/_stats/fielddata/{fields}'
]
};

View File

@ -1,14 +1,16 @@
module.exports = function (done) {
module.exports = function (branch, done) {
/**
* Creates a JSON version of the YAML test suite that can be simply bundled for use in the browser.
*/
var jsYaml = require('js-yaml');
var fs = require('relative-fs').relativeTo(__dirname);
var async = require('async');
var _ = require('../../src/lib/utils');
var path = require('path');
var tests = {}; // populated in readYamlTests
var branchSuffix = branch === 'master' ? '' : '_' + _.snakeCase(branch);
// generate the yaml tests
async.series([
readYamlTests,
@ -38,7 +40,7 @@ module.exports = function (done) {
}
function writeYamlTests(done) {
var testFile = require('path').resolve(__dirname, '../../test/integration/yaml_suite/yaml_tests.json');
var testFile = require('path').resolve(__dirname, '../../test/integration/yaml_suite/yaml_tests' + branchSuffix + '.json');
fs.writeFileSync(testFile, JSON.stringify(tests, null, ' '), 'utf8');
console.log('wrote YAML tests as JSON to', testFile);
done();

View File

@ -1545,8 +1545,8 @@ api.indices.prototype['delete'] = ca({
* @param {Object} params - An object with parameters used to carry out this action
* @param {Date, Number} params.timeout - Explicit timestamp for the document
* @param {Date, Number} params.masterTimeout - Specify timeout for connection to master
* @param {String} params.index - The name of the index with an alias
* @param {String} params.name - The name of the alias to be deleted
* @param {String, String[], Boolean} params.index - A comma-separated list of index names (supports wildcards); use `_all` for all indices
* @param {String, String[], Boolean} params.name - A comma-separated list of aliases to delete (supports wildcards); use `_all` to delete all aliases for the specified indices.
*/
api.indices.prototype.deleteAlias = ca({
params: {
@ -1558,30 +1558,18 @@ api.indices.prototype.deleteAlias = ca({
name: 'master_timeout'
}
},
urls: [
{
fmt: '/<%=index%>/_alias/<%=name%>',
req: {
index: {
type: 'string'
},
name: {
type: 'string'
}
url: {
fmt: '/<%=index%>/_alias/<%=name%>',
req: {
index: {
type: 'list'
},
name: {
type: 'list'
}
},
{
fmt: '/<%=index%>/_aliases/<%=name%>',
req: {
index: {
type: 'string'
},
name: {
type: 'string'
}
}
}
],
sortOrder: -2
},
method: 'DELETE'
});
@ -1590,8 +1578,8 @@ api.indices.prototype.deleteAlias = ca({
*
* @param {Object} params - An object with parameters used to carry out this action
* @param {Date, Number} params.masterTimeout - Specify timeout for connection to master
* @param {String, String[], Boolean} params.index - A comma-separated list of index names; use `_all` for all indices
* @param {String} params.type - The name of the document type to delete
* @param {String, String[], Boolean} params.index - A comma-separated list of index names (supports wildcards); use `_all` for all indices
* @param {String, String[], Boolean} params.type - A comma-separated list of document types to delete (supports wildcards); use `_all` to delete all document types in the specified indices.
*/
api.indices.prototype.deleteMapping = ca({
params: {
@ -1600,52 +1588,18 @@ api.indices.prototype.deleteMapping = ca({
name: 'master_timeout'
}
},
urls: [
{
fmt: '/<%=index%>/<%=type%>',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
url: {
fmt: '/<%=index%>/<%=type%>/_mapping',
req: {
index: {
type: 'list'
},
type: {
type: 'list'
}
},
{
fmt: '/<%=index%>/_mapping/<%=type%>',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/<%=type%>/_mappings',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/_mappings/<%=type%>',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
}
],
sortOrder: -2
},
method: 'DELETE'
});
@ -1684,15 +1638,17 @@ api.indices.prototype.deleteTemplate = ca({
*
* @param {Object} params - An object with parameters used to carry out this action
* @param {Date, Number} params.masterTimeout - Specify timeout for connection to master
* @param {String, String[], Boolean} params.index - A comma-separated list of index names to register warmer for; use `_all` or empty string to perform the operation on all indices
* @param {String} params.name - The name of the warmer (supports wildcards); leave empty to delete all warmers
* @param {String, String[], Boolean} params.type - A comma-separated list of document types to register warmer for; use `_all` or empty string to perform the operation on all types
* @param {String, String[], Boolean} params.name - A comma-separated list of warmer names to delete (supports wildcards); use `_all` to delete all warmers in the specified indices. You must specify a name either in the uri or in the parameters.
* @param {String, String[], Boolean} params.index - A comma-separated list of index names to delete warmers from (supports wildcards); use `_all` to perform the operation on all indices.
*/
api.indices.prototype.deleteWarmer = ca({
params: {
masterTimeout: {
type: 'time',
name: 'master_timeout'
},
name: {
type: 'list'
}
},
url: {
@ -1702,7 +1658,7 @@ api.indices.prototype.deleteWarmer = ca({
type: 'list'
},
name: {
type: 'string'
type: 'list'
}
},
sortOrder: -2
@ -2493,7 +2449,7 @@ api.indices.prototype.optimize = ca({
* @param {Object} params - An object with parameters used to carry out this action
* @param {Date, Number} params.timeout - Explicit timestamp for the document
* @param {Date, Number} params.masterTimeout - Specify timeout for connection to master
* @param {String} params.index - The name of the index with an alias
* @param {String, String[], Boolean} params.index - A comma-separated list of index names the alias should point to (supports wildcards); use `_all` or omit to perform the operation on all indices.
* @param {String} params.name - The name of the alias to be created or updated
*/
api.indices.prototype.putAlias = ca({
@ -2511,18 +2467,7 @@ api.indices.prototype.putAlias = ca({
fmt: '/<%=index%>/_alias/<%=name%>',
req: {
index: {
type: 'string'
},
name: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/_aliases/<%=name%>',
req: {
index: {
type: 'string'
type: 'list'
},
name: {
type: 'string'
@ -2536,14 +2481,6 @@ api.indices.prototype.putAlias = ca({
type: 'string'
}
}
},
{
fmt: '/_aliases/<%=name%>',
req: {
name: {
type: 'string'
}
}
}
],
method: 'PUT'
@ -2559,7 +2496,7 @@ api.indices.prototype.putAlias = ca({
* @param {Boolean} params.ignoreUnavailable - Whether specified concrete indices should be ignored when unavailable (missing or closed)
* @param {Boolean} params.allowNoIndices - Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes `_all` string or when no indices have been specified)
* @param {String} [params.expandWildcards=open] - Whether to expand wildcard expression to concrete indices that are open, closed or both.
* @param {String, String[], Boolean} params.index - A comma-separated list of index names; use `_all` to perform the operation on all indices
* @param {String, String[], Boolean} params.index - A comma-separated list of index names the mapping should be added to (supports wildcards); use `_all` or omit to add the mapping on all indices.
* @param {String} params.type - The name of the document type
*/
api.indices.prototype.putMapping = ca({
@ -2594,17 +2531,6 @@ api.indices.prototype.putMapping = ca({
}
},
urls: [
{
fmt: '/<%=index%>/<%=type%>/_mapping',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/_mapping/<%=type%>',
req: {
@ -2616,28 +2542,6 @@ api.indices.prototype.putMapping = ca({
}
}
},
{
fmt: '/<%=index%>/<%=type%>/_mappings',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/_mappings/<%=type%>',
req: {
index: {
type: 'list'
},
type: {
type: 'string'
}
}
},
{
fmt: '/_mapping/<%=type%>',
req: {
@ -2645,14 +2549,6 @@ api.indices.prototype.putMapping = ca({
type: 'string'
}
}
},
{
fmt: '/_mappings/<%=type%>',
req: {
type: {
type: 'string'
}
}
}
],
method: 'PUT'
@ -2760,7 +2656,7 @@ api.indices.prototype.putTemplate = ca({
* @param {Boolean} params.ignoreUnavailable - Whether specified concrete indices should be ignored when unavailable (missing or closed) in the search request to warm
* @param {Boolean} params.allowNoIndices - Whether to ignore if a wildcard indices expression resolves into no concrete indices in the search request to warm. (This includes `_all` string or when no indices have been specified)
* @param {String} [params.expandWildcards=open] - Whether to expand wildcard expression to concrete indices that are open, closed or both, in the search request to warm.
* @param {String, String[], Boolean} params.index - A comma-separated list of index names to register the warmer for; use `_all` or empty string to perform the operation on all indices
* @param {String, String[], Boolean} params.index - A comma-separated list of index names to register the warmer for; use `_all` or omit to perform the operation on all indices
* @param {String} params.name - The name of the warmer
* @param {String, String[], Boolean} params.type - A comma-separated list of document types to register the warmer for; leave empty to perform the operation on all types
*/
@ -2803,20 +2699,6 @@ api.indices.prototype.putWarmer = ca({
}
}
},
{
fmt: '/<%=index%>/<%=type%>/_warmers/<%=name%>',
req: {
index: {
type: 'list'
},
type: {
type: 'list'
},
name: {
type: 'string'
}
}
},
{
fmt: '/<%=index%>/_warmer/<%=name%>',
req: {
@ -2828,17 +2710,6 @@ api.indices.prototype.putWarmer = ca({
}
}
},
{
fmt: '/<%=index%>/_warmers/<%=name%>',
req: {
index: {
type: 'list'
},
name: {
type: 'string'
}
}
},
{
fmt: '/_warmer/<%=name%>',
req: {
@ -2846,14 +2717,6 @@ api.indices.prototype.putWarmer = ca({
type: 'string'
}
}
},
{
fmt: '/_warmers/<%=name%>',
req: {
name: {
type: 'string'
}
}
}
],
method: 'PUT'

3226
src/lib/api_0_90.js Normal file

File diff suppressed because it is too large Load Diff

View File

@ -26,50 +26,57 @@
module.exports = Client;
var api = require('./api.js');
var ca = require('./client_action');
var Transport = require('./transport');
var ca = require('./client_action');
var _ = require('./utils');
var defaultApi = 'master';
var apis = {
'0.90': require('./api_0_90'),
'master': require('./api')
};
function Client(config) {
if (!(this instanceof Client)) {
return new Client(config);
}
config = config || {};
// our client will log minimally by default
if (!config.hasOwnProperty('log')) {
config.log = 'warning';
function EsApiClient(config) {
// our client will log minimally by default
if (!config.hasOwnProperty('log')) {
config.log = 'warning';
}
if (!config.hosts && !config.host) {
config.host = 'http://localhost:9200';
}
this.ping = ca({
method: 'HEAD',
url: {
fmt: '/'
},
castExists: true,
requestTimeout: 100
});
this.close = function () {
this.transport.close();
};
this.transport = new Transport(config);
// instantiate the api's namespaces
for (var i = 0; i < this._namespaces.length; i++) {
this[this._namespaces[i]] = new this[this._namespaces[i]](this.transport);
}
delete this._namespaces;
}
if (!config.hosts && !config.host) {
config.host = 'http://localhost:9200';
var apiVersion = config.apiVersion || defaultApi;
if (apis.hasOwnProperty(apiVersion)) {
EsApiClient.prototype = apis[apiVersion];
} else {
throw new Error('Invalid apiVersion "' + apiVersion + '", expected one of ' + _.keys(apis).join(', '));
}
this.transport = new Transport(config);
// instantiate the api's namespaces
for (var i = 0; i < this._namespaces.length; i++) {
this[this._namespaces[i]] = new this[this._namespaces[i]](this.transport);
}
return new EsApiClient(config);
}
Client.prototype = api;
/**
* Ping some node to ensure that the cluster is available in some respect
*
* @param {Object} params - Currently just a placeholder, no params used at this time
* @param {Function} cb - callback
*/
Client.prototype.ping = ca({
method: 'HEAD',
url: {
fmt: '/'
},
castExists: true,
requestTimeout: 100
});
Client.prototype.close = function () {
this.transport.close();
};

View File

@ -100,7 +100,7 @@ function adjustWordCase(firstWordCap, otherWordsCap, sep) {
for (; i < string.length; i++) {
code = string.charCodeAt(i);
c = string.charAt(i);
lower = code >= 97 && code <= 122;
lower = (code >= 97 && code <= 122) || (code >= 48 && code <= 57);
upper = code >= 65 && code <= 90;
if (upper || !lower) {

View File

@ -8,7 +8,6 @@ if (BROWSER) {
var es = require('../../../src/elasticsearch');
}
var argv = require('./argv');
var server = require('./server');
var fs = require('relative-fs').relativeTo(require('path').join(__dirname, '../../../'));
var _ = require('../../../src/lib/utils');
@ -18,11 +17,8 @@ var client = null;
// when set to a boolean, hold the test of a ping
var externalExists;
// a reference to a personal instance of ES Server
var esServer = null;
module.exports = {
create: function create(cb) {
create: function create(branch, cb) {
// create a client and ping the server for up to 15 seconds
doCreateClient({
logConfig: null
@ -31,14 +27,17 @@ module.exports = {
var timeout = 500;
(function ping() {
client.ping({
client.info({
maxRetries: 0,
requestTimeout: 100
}, function (err) {
}, function (err, resp) {
if (err && --attemptsRemaining) {
setTimeout(ping, timeout);
} else if (err) {
cb(new Error('unable to establish contact with ES'));
} else if (resp.name !== 'elasticsearch_js_test_runner') {
console.log(resp);
cb(new Error('Almosted wiped out another es node. Shut-down all instances of ES and try again.'));
} else {
// create a new client
doCreateClient(function () {
@ -79,10 +78,11 @@ module.exports = {
}
client = new es.Client({
apiVersion: branch,
hosts: [
{
host: esServer ? esServer.__hostname : argv.host,
port: esServer ? esServer.__port : argv.port
host: argv.host,
port: argv.port
}
],
log: logConfig

View File

@ -1,33 +1 @@
var path = require('path');
var async = require('async');
var jsYaml = require('js-yaml');
var YamlFile = require('./yaml_file');
var _ = require('../../../src/lib/utils');
var es = require('../../../src/elasticsearch');
var clientManager = require('./client_manager');
var argv = require('./argv');
var testDir = path.resolve(__dirname, './tests');
describe('integration', function () {
this.timeout(30000);
// before running any tests...
before(function (done) {
// start our personal ES Server
this.timeout(null);
clientManager.create(done);
});
before(function (done) {
// make sure ES is empty
clientManager.get().indices.delete({
index: '*',
ignore: 404
}, done);
});
var files = _.map(require('./yaml_tests.json'), function (docs, filename) {
return new YamlFile(filename, docs);
});
});
require('./run')('master');

View File

@ -0,0 +1 @@
require('./run')('0.90');

View File

@ -0,0 +1,35 @@
module.exports = function (branch) {
var path = require('path');
var async = require('async');
var jsYaml = require('js-yaml');
var YamlFile = require('./yaml_file');
var _ = require('../../../src/lib/utils');
var es = require('../../../src/elasticsearch');
var clientManager = require('./client_manager');
var argv = require('./argv');
var branchSuffix = branch === 'master' ? '' : '_' + _.snakeCase(branch);
describe('integration', function () {
this.timeout(30000);
// before running any tests...
before(function (done) {
// start our personal ES Server
this.timeout(null);
clientManager.create(branch, done);
});
before(function (done) {
// make sure ES is empty
clientManager.get().indices.delete({
index: '*',
ignore: 404
}, done);
});
var files = _.map(require('./yaml_tests' + branchSuffix + '.json'), function (docs, filename) {
return new YamlFile(filename, docs);
});
});
};

View File

@ -1,65 +0,0 @@
var childProc = require('child_process');
var events = require('events');
var path = require('path');
var fs = require('fs');
var _ = require('../../../src/lib/utils');
var argv = require('./argv');
exports.start = function (cb) {
if (!argv.executable || !fs.existsSync(argv.executable)) {
return cb(new Error('unable to find elasticsearch executable, ' +
'set ES_HOME env var to the instalation path of elasticsearch'));
}
var server = childProc.spawn(
argv.executable,
[
'-f',
'-Des.cluster.name=' + argv.clusterName,
'-Des.path.data=' + argv.dataPath,
// '-Des.logger.level=DEBUG',
'-Des.discovery.zen.ping.multicast.enabled=false',
],
{
cwd: void 0,
env: process.env,
stdio: [
'ignore',
'pipe',
'pipe'
]
}
);
server.stdout.on('data', function onProcessData(line) {
line = line.toString().trim();
var match;
if (match = line.match(/\{inet\[\/?([^:]+):(\d+)\]\}/)) {
server.__hostname = match[1];
server.__port = match[2];
}
if (line.match(/started\s*$/m)) {
console.log('Personal ES Server started at', server.__hostname + ':' + server.__port);
server.stdout.removeListener('data', onProcessData);
server.stdout.resume();
cb(null, server);
}
});
server.stderr.on('data', function (line) {
console.error(line.toString().trim());
});
server.on('close', function (code) {
server.stdout.removeAllListeners();
server.stderr.removeAllListeners();
console.log('Personal ES Server Shutdown with exit code', code);
});
process.on('exit', function () {
server.kill();
});
};

File diff suppressed because it is too large Load Diff

View File

@ -57,7 +57,7 @@ describe('Http Connector', function () {
});
it('expects the host to have a protocol of http or https', function () {
(function () {
expect(function () {
var con = new HttpConnection(new Host('thrifty://es.com/stuff'));
}).to.throwError(/invalid protocol/i);
});
@ -172,9 +172,9 @@ describe('Http Connector', function () {
it('calls http based on the host', function (done) {
var con = new HttpConnection(new Host('http://google.com'));
con.request({}, function () {
http.request.callCount.should.eql(1);
https.request.callCount.should.eql(0);
http.request.lastCall.args[0].agent.should.be.an.instanceOf(KeepAliveAgent);
expect(http.request.callCount).to.be(1);
expect(https.request.callCount).to.be(0);
expect(http.request.lastCall.args[0].agent).to.be.a(KeepAliveAgent);
done();
});
});
@ -182,9 +182,9 @@ describe('Http Connector', function () {
it('calls https based on the host', function (done) {
var con = new HttpConnection(new Host('https://google.com'));
con.request({}, function () {
http.request.callCount.should.eql(0);
https.request.callCount.should.eql(1);
https.request.lastCall.args[0].agent.should.be.an.instanceOf(KeepAliveAgent.HttpsAgent);
expect(http.request.callCount).to.be(0);
expect(https.request.callCount).to.be(1);
expect(https.request.lastCall.args[0].agent).to.be.a(KeepAliveAgent.HttpsAgent);
done();
});
});

View File

@ -138,6 +138,10 @@ describe('Utils', function () {
it('handles leading _', function () {
expect(_.camelCase('_thing_one_')).to.eql('_thingOne');
});
it('works on numbers', function () {
expect(_.camelCase('version 1.0')).to.eql('version10');
});
});
describe('#studlyCase', function () {
@ -152,6 +156,10 @@ describe('Utils', function () {
it('handles leading _', function () {
expect(_.studlyCase('_thing_one_')).to.eql('_ThingOne');
});
it('works on numbers', function () {
expect(_.studlyCase('version 1.0')).to.eql('Version10');
});
});
describe('#snakeCase', function () {
@ -166,6 +174,10 @@ describe('Utils', function () {
it('handles leading _', function () {
expect(_.snakeCase('_thing_one_')).to.eql('_thing_one');
});
it('works on numbers', function () {
expect(_.snakeCase('version 1.0')).to.eql('version_1_0');
});
});
describe('#toLowerString', function () {